{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "966f3616",
   "metadata": {},
   "source": [
    "# 文本分类"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "5b9db8a6",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "matplotlib version: 3.10.3\n",
      "numpy version: 2.0.2\n",
      "tiktoken version: 0.9.0\n",
      "torch version: 2.7.1\n",
      "tensorflow version: 2.19.0\n",
      "pandas version: 2.3.1\n"
     ]
    }
   ],
   "source": [
    "from importlib.metadata import version\n",
    "\n",
    "pkgs = [\"matplotlib\",  # Plotting library\n",
    "        \"numpy\",       # PyTorch & TensorFlow dependency\n",
    "        \"tiktoken\",    # Tokenizer\n",
    "        \"torch\",       # Deep learning library\n",
    "        \"tensorflow\",  # For OpenAI's pretrained weights\n",
    "        \"pandas\"       # Dataset loading\n",
    "       ]\n",
    "for p in pkgs:\n",
    "    print(f\"{p} version: {version(p)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "de15b1a3",
   "metadata": {},
   "source": [
    "![](./images/chapter-overview.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aea234ec",
   "metadata": {},
   "source": [
    "## 微调的不同类别"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fe663be6",
   "metadata": {},
   "source": [
    "- 微调语言模型最常见的方式是指令微调和分类微调；\n",
    "- 下图展示的指令微调将是下一章的主题；"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cae619ab",
   "metadata": {},
   "source": [
    "![](./images/instructions.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0105fe02",
   "metadata": {},
   "source": [
    "- 分类微调（本章主题）对于有机器学习背景的读者可能并不陌生——例如，这类似于训练卷积网络对手写数字进行分类；\n",
    "- 在分类微调中，模型可输出的类别标签数量是固定的（例如“垃圾邮件”和“非垃圾邮件”）；\n",
    "- 经过分类微调的模型只能预测训练时见过的类别（如“垃圾邮件”或“非垃圾邮件”），而指令微调模型通常能执行多种任务；\n",
    "- 我们可以将分类微调模型视为高度专业化的模型；实践中，创建专业化模型比构建能胜任多种任务的通用模型要容易得多；\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "887b6251",
   "metadata": {},
   "source": [
    "![](./images/spam-non-spam.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1a2255bf",
   "metadata": {},
   "source": [
    "## 准备数据"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "289c16b3",
   "metadata": {},
   "source": [
    "- 本节将准备用于分类微调的数据集；\n",
    "- 我们将使用包含垃圾邮件与非垃圾邮件的短信数据集对大语言模型进行分类微调；\n",
    "- 首先下载并解压缩数据集；\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "453487b3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "sms_spam_collection/SMSSpamCollection.tsv already exists. Skipping download and extraction.\n"
     ]
    }
   ],
   "source": [
    "import urllib.request\n",
    "import zipfile\n",
    "import os\n",
    "from pathlib import Path\n",
    "\n",
    "url = \"https://archive.ics.uci.edu/static/public/228/sms+spam+collection.zip\"\n",
    "zip_path = \"sms_spam_collection.zip\"\n",
    "extracted_path = \"sms_spam_collection\"\n",
    "data_file_path = Path(extracted_path) / \"SMSSpamCollection.tsv\"\n",
    "\n",
    "def download_and_unzip_spam_data(url, zip_path, extracted_path, data_file_path):\n",
    "    if data_file_path.exists():\n",
    "        print(f\"{data_file_path} already exists. Skipping download and extraction.\")\n",
    "        return\n",
    "\n",
    "    # Downloading the file\n",
    "    with urllib.request.urlopen(url) as response:\n",
    "        with open(zip_path, \"wb\") as out_file:\n",
    "            out_file.write(response.read())\n",
    "\n",
    "    # Unzipping the file\n",
    "    with zipfile.ZipFile(zip_path, \"r\") as zip_ref:\n",
    "        zip_ref.extractall(extracted_path)\n",
    "\n",
    "    # Add .tsv file extension\n",
    "    original_file_path = Path(extracted_path) / \"SMSSpamCollection\"\n",
    "    os.rename(original_file_path, data_file_path)\n",
    "    print(f\"File downloaded and saved as {data_file_path}\")\n",
    "\n",
    "try:\n",
    "    download_and_unzip_spam_data(url, zip_path, extracted_path, data_file_path)\n",
    "except (urllib.error.HTTPError, urllib.error.URLError, TimeoutError) as e:\n",
    "    print(f\"Primary URL failed: {e}. Trying backup URL...\")\n",
    "    url = \"https://f001.backblazeb2.com/file/LLMs-from-scratch/sms%2Bspam%2Bcollection.zip\"\n",
    "    download_and_unzip_spam_data(url, zip_path, extracted_path, data_file_path) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "1a0be079",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Label</th>\n",
       "      <th>Text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>ham</td>\n",
       "      <td>Go until jurong point, crazy.. Available only ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>ham</td>\n",
       "      <td>Ok lar... Joking wif u oni...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>spam</td>\n",
       "      <td>Free entry in 2 a wkly comp to win FA Cup fina...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>ham</td>\n",
       "      <td>U dun say so early hor... U c already then say...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>ham</td>\n",
       "      <td>Nah I don't think he goes to usf, he lives aro...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5567</th>\n",
       "      <td>spam</td>\n",
       "      <td>This is the 2nd time we have tried 2 contact u...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5568</th>\n",
       "      <td>ham</td>\n",
       "      <td>Will ü b going to esplanade fr home?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5569</th>\n",
       "      <td>ham</td>\n",
       "      <td>Pity, * was in mood for that. So...any other s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5570</th>\n",
       "      <td>ham</td>\n",
       "      <td>The guy did some bitching but I acted like i'd...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5571</th>\n",
       "      <td>ham</td>\n",
       "      <td>Rofl. Its true to its name</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5572 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     Label                                               Text\n",
       "0      ham  Go until jurong point, crazy.. Available only ...\n",
       "1      ham                      Ok lar... Joking wif u oni...\n",
       "2     spam  Free entry in 2 a wkly comp to win FA Cup fina...\n",
       "3      ham  U dun say so early hor... U c already then say...\n",
       "4      ham  Nah I don't think he goes to usf, he lives aro...\n",
       "...    ...                                                ...\n",
       "5567  spam  This is the 2nd time we have tried 2 contact u...\n",
       "5568   ham               Will ü b going to esplanade fr home?\n",
       "5569   ham  Pity, * was in mood for that. So...any other s...\n",
       "5570   ham  The guy did some bitching but I acted like i'd...\n",
       "5571   ham                         Rofl. Its true to its name\n",
       "\n",
       "[5572 rows x 2 columns]"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "df = pd.read_csv(data_file_path, sep=\"\\t\", header=None, names=[\"Label\", \"Text\"])\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "53e7d0d2",
   "metadata": {},
   "source": [
    "- 为简化流程且适配教学场景对小数据集的需求（这将加速大语言模型的微调过程），我们对数据集进行了子采样（下采样），使每个类别仅保留747个实例；\n",
    "- 除下采样外，还存在其他处理类别平衡的方法，但这已超出大语言模型教材的讨论范围；\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "id": "30ca79b8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Label\n",
      "ham     747\n",
      "spam    747\n",
      "Name: count, dtype: int64\n"
     ]
    }
   ],
   "source": [
    "def create_balanced_dataset(df):\n",
    "    \n",
    "    # Count the instances of \"spam\"\n",
    "    num_spam = df[df[\"Label\"] == \"spam\"].shape[0]\n",
    "    \n",
    "    # Randomly sample \"ham\" instances to match the number of \"spam\" instances\n",
    "    ham_subset = df[df[\"Label\"] == \"ham\"].sample(num_spam, random_state=123)\n",
    "    \n",
    "    # Combine ham \"subset\" with \"spam\"\n",
    "    balanced_df = pd.concat([ham_subset, df[df[\"Label\"] == \"spam\"]])\n",
    "\n",
    "    return balanced_df\n",
    "\n",
    "\n",
    "balanced_df = create_balanced_dataset(df)\n",
    "print(balanced_df[\"Label\"].value_counts())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "id": "267b68cd",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Label</th>\n",
       "      <th>Text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>4307</th>\n",
       "      <td>0</td>\n",
       "      <td>Awww dat is sweet! We can think of something t...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4138</th>\n",
       "      <td>0</td>\n",
       "      <td>Just got to  &amp;lt;#&amp;gt;</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4831</th>\n",
       "      <td>0</td>\n",
       "      <td>The word \"Checkmate\" in chess comes from the P...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4461</th>\n",
       "      <td>0</td>\n",
       "      <td>This is wishing you a great day. Moji told me ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5440</th>\n",
       "      <td>0</td>\n",
       "      <td>Thank you. do you generally date the brothas?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5537</th>\n",
       "      <td>1</td>\n",
       "      <td>Want explicit SEX in 30 secs? Ring 02073162414...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5540</th>\n",
       "      <td>1</td>\n",
       "      <td>ASKED 3MOBILE IF 0870 CHATLINES INCLU IN FREE ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5547</th>\n",
       "      <td>1</td>\n",
       "      <td>Had your contract mobile 11 Mnths? Latest Moto...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5566</th>\n",
       "      <td>1</td>\n",
       "      <td>REMINDER FROM O2: To get 2.50 pounds free call...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5567</th>\n",
       "      <td>1</td>\n",
       "      <td>This is the 2nd time we have tried 2 contact u...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1494 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      Label                                               Text\n",
       "4307      0  Awww dat is sweet! We can think of something t...\n",
       "4138      0                             Just got to  &lt;#&gt;\n",
       "4831      0  The word \"Checkmate\" in chess comes from the P...\n",
       "4461      0  This is wishing you a great day. Moji told me ...\n",
       "5440      0      Thank you. do you generally date the brothas?\n",
       "...     ...                                                ...\n",
       "5537      1  Want explicit SEX in 30 secs? Ring 02073162414...\n",
       "5540      1  ASKED 3MOBILE IF 0870 CHATLINES INCLU IN FREE ...\n",
       "5547      1  Had your contract mobile 11 Mnths? Latest Moto...\n",
       "5566      1  REMINDER FROM O2: To get 2.50 pounds free call...\n",
       "5567      1  This is the 2nd time we have tried 2 contact u...\n",
       "\n",
       "[1494 rows x 2 columns]"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "balanced_df[\"Label\"] = balanced_df[\"Label\"].map({\"ham\": 0, \"spam\": 1}) \n",
    "balanced_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "id": "0187330d",
   "metadata": {},
   "outputs": [],
   "source": [
    "def random_split(df, train_frac, validation_frac):\n",
    "    # Shuffle the entire DataFrame\n",
    "    df = df.sample(frac=1, random_state=123).reset_index(drop=True)\n",
    "\n",
    "    # Calculate split indices\n",
    "    train_end = int(len(df) * train_frac)\n",
    "    validation_end = train_end + int(len(df) * validation_frac)\n",
    "\n",
    "    # Split the DataFrame\n",
    "    train_df = df[:train_end]\n",
    "    validation_df = df[train_end:validation_end]\n",
    "    test_df = df[validation_end:]\n",
    "\n",
    "    return train_df, validation_df, test_df\n",
    "\n",
    "train_df, validation_df, test_df = random_split(balanced_df, 0.7, 0.1)\n",
    "# Test size is implied to be 0.2 as the remainder\n",
    "\n",
    "train_df.to_csv(\"train.csv\", index=None)\n",
    "validation_df.to_csv(\"validation.csv\", index=None)\n",
    "test_df.to_csv(\"test.csv\", index=None)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e1d1ee30",
   "metadata": {},
   "source": [
    "### 创建Dataloader"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "af11f0e6",
   "metadata": {},
   "source": [
    "- 请注意短信文本长度不一；若希望将多个训练样本组合成批量数据，则必须选择以下方式之一：\n",
    "    - 将所有短信截断至数据集中最短样本的长度；\n",
    "    - 将所有短信填充至数据集中最长样本的长度；\n",
    "\n",
    "- 我们选择方案2，将所有短信填充至数据集中的最大长度；\n",
    "- 根据第2章讨论，我们使用<|endoftext|>作为填充符；"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3bb79fb6",
   "metadata": {},
   "source": [
    "![](./images/pad-input-sequences.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "id": "0cfd7df6",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[50256]\n"
     ]
    }
   ],
   "source": [
    "import tiktoken\n",
    "\n",
    "tokenizer = tiktoken.get_encoding(\"gpt2\")\n",
    "print(tokenizer.encode(\"<|endoftext|>\", allowed_special={\"<|endoftext|>\"}))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "id": "51139ea3",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "from torch.utils.data import Dataset\n",
    "\n",
    "\n",
    "class SpamDataset(Dataset):\n",
    "    def __init__(self, csv_file, tokenizer, max_length=None, pad_token_id=50256):\n",
    "        self.data = pd.read_csv(csv_file)\n",
    "\n",
    "        # Pre-tokenize texts\n",
    "        self.encoded_texts = [\n",
    "            tokenizer.encode(text) for text in self.data[\"Text\"]\n",
    "        ]\n",
    "\n",
    "        if max_length is None:\n",
    "            self.max_length = self._longest_encoded_length()\n",
    "        else:\n",
    "            self.max_length = max_length\n",
    "            # Truncate sequences if they are longer than max_length\n",
    "            self.encoded_texts = [\n",
    "                encoded_text[:self.max_length]\n",
    "                for encoded_text in self.encoded_texts\n",
    "            ]\n",
    "\n",
    "        # Pad sequences to the longest sequence\n",
    "        self.encoded_texts = [\n",
    "            encoded_text + [pad_token_id] * (self.max_length - len(encoded_text))\n",
    "            for encoded_text in self.encoded_texts\n",
    "        ]\n",
    "\n",
    "    def __getitem__(self, index):\n",
    "        encoded = self.encoded_texts[index]\n",
    "        label = self.data.iloc[index][\"Label\"]\n",
    "        return (\n",
    "            torch.tensor(encoded, dtype=torch.long),\n",
    "            torch.tensor(label, dtype=torch.long)\n",
    "        )\n",
    "\n",
    "    def __len__(self):\n",
    "        return len(self.data)\n",
    "\n",
    "    def _longest_encoded_length(self):\n",
    "        max_length = 0\n",
    "        for encoded_text in self.encoded_texts:\n",
    "            encoded_length = len(encoded_text)\n",
    "            if encoded_length > max_length:\n",
    "                max_length = encoded_length\n",
    "        return max_length\n",
    "        # Note: A more pythonic version to implement this method\n",
    "        # is the following, which is also used in the next chapter:\n",
    "        # return max(len(encoded_text) for encoded_text in self.encoded_texts)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "id": "8e7b546a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "120\n"
     ]
    }
   ],
   "source": [
    "train_dataset = SpamDataset(\n",
    "    csv_file=\"train.csv\",\n",
    "    max_length=None,\n",
    "    tokenizer=tokenizer\n",
    ")\n",
    "\n",
    "print(train_dataset.max_length)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a663ce31",
   "metadata": {},
   "source": [
    "- 验证集与测试集同样按训练集的最长序列长度进行填充；\n",
    "- 注意：若验证集或测试集中存在长于训练集最长样本的序列，会通过SpamDataset代码中的encoded_text[:self.max_length]进行截断；\n",
    "- 此处理方式并非强制，将验证集与测试集的max_length参数设为None同样可行；\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "id": "8e0703f3",
   "metadata": {},
   "outputs": [],
   "source": [
    "val_dataset = SpamDataset(\n",
    "    csv_file=\"validation.csv\",\n",
    "    max_length=train_dataset.max_length,\n",
    "    tokenizer=tokenizer\n",
    ")\n",
    "test_dataset = SpamDataset(\n",
    "    csv_file=\"test.csv\",\n",
    "    max_length=train_dataset.max_length,\n",
    "    tokenizer=tokenizer\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d09abf87",
   "metadata": {},
   "source": [
    "![](./images/batch.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "id": "e2983f53",
   "metadata": {},
   "outputs": [],
   "source": [
    "from torch.utils.data import DataLoader\n",
    "\n",
    "num_workers = 0\n",
    "batch_size = 8\n",
    "\n",
    "torch.manual_seed(123)\n",
    "\n",
    "train_loader = DataLoader(\n",
    "    dataset=train_dataset,\n",
    "    batch_size=batch_size,\n",
    "    shuffle=True,\n",
    "    num_workers=num_workers,\n",
    "    drop_last=True,\n",
    ")\n",
    "\n",
    "val_loader = DataLoader(\n",
    "    dataset=val_dataset,\n",
    "    batch_size=batch_size,\n",
    "    num_workers=num_workers,\n",
    "    drop_last=False,\n",
    ")\n",
    "\n",
    "test_loader = DataLoader(\n",
    "    dataset=test_dataset,\n",
    "    batch_size=batch_size,\n",
    "    num_workers=num_workers,\n",
    "    drop_last=False,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "id": "6d8a48e1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train loader:\n",
      "Input batch dimensions: torch.Size([8, 120])\n",
      "Label batch dimensions torch.Size([8])\n"
     ]
    }
   ],
   "source": [
    "print(\"Train loader:\")\n",
    "for input_batch, target_batch in train_loader:\n",
    "    pass\n",
    "\n",
    "print(\"Input batch dimensions:\", input_batch.shape)\n",
    "print(\"Label batch dimensions\", target_batch.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "id": "970717d9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "130 training batches\n",
      "19 validation batches\n",
      "38 test batches\n"
     ]
    }
   ],
   "source": [
    "print(f\"{len(train_loader)} training batches\")\n",
    "print(f\"{len(val_loader)} validation batches\")\n",
    "print(f\"{len(test_loader)} test batches\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c23d02ef",
   "metadata": {},
   "source": [
    "## 加载权重"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dc004ce8",
   "metadata": {},
   "source": [
    "![](./images/overview-2.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "id": "ee9021c7",
   "metadata": {},
   "outputs": [],
   "source": [
    "CHOOSE_MODEL = \"gpt2-small (124M)\"\n",
    "INPUT_PROMPT = \"Every effort moves\"\n",
    "\n",
    "BASE_CONFIG = {\n",
    "    \"vocab_size\": 50257,     # Vocabulary size\n",
    "    \"context_length\": 1024,  # Context length\n",
    "    \"drop_rate\": 0.0,        # Dropout rate\n",
    "    \"qkv_bias\": True         # Query-key-value bias\n",
    "}\n",
    "\n",
    "model_configs = {\n",
    "    \"gpt2-small (124M)\": {\"emb_dim\": 768, \"n_layers\": 12, \"n_heads\": 12},\n",
    "    \"gpt2-medium (355M)\": {\"emb_dim\": 1024, \"n_layers\": 24, \"n_heads\": 16},\n",
    "    \"gpt2-large (774M)\": {\"emb_dim\": 1280, \"n_layers\": 36, \"n_heads\": 20},\n",
    "    \"gpt2-xl (1558M)\": {\"emb_dim\": 1600, \"n_layers\": 48, \"n_heads\": 25},\n",
    "}\n",
    "\n",
    "BASE_CONFIG.update(model_configs[CHOOSE_MODEL])\n",
    "\n",
    "assert train_dataset.max_length <= BASE_CONFIG[\"context_length\"], (\n",
    "    f\"Dataset length {train_dataset.max_length} exceeds model's context \"\n",
    "    f\"length {BASE_CONFIG['context_length']}. Reinitialize data sets with \"\n",
    "    f\"`max_length={BASE_CONFIG['context_length']}`\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "id": "fd4500fe",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "File already exists and is up-to-date: gpt2/124M/checkpoint\n",
      "File already exists and is up-to-date: gpt2/124M/encoder.json\n",
      "File already exists and is up-to-date: gpt2/124M/hparams.json\n",
      "File already exists and is up-to-date: gpt2/124M/model.ckpt.data-00000-of-00001\n",
      "File already exists and is up-to-date: gpt2/124M/model.ckpt.index\n",
      "File already exists and is up-to-date: gpt2/124M/model.ckpt.meta\n",
      "File already exists and is up-to-date: gpt2/124M/vocab.bpe\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2025-10-06 09:22:29.274977: W external/local_xla/xla/tsl/framework/cpu_allocator_impl.cc:83] Allocation of 154389504 exceeds 10% of free system memory.\n"
     ]
    }
   ],
   "source": [
    "from gpt_download import download_and_load_gpt2\n",
    "from utils import GPTModel, load_weights_into_gpt\n",
    "\n",
    "model_size = CHOOSE_MODEL.split(\" \")[-1].lstrip(\"(\").rstrip(\")\")\n",
    "settings, params = download_and_load_gpt2(model_size=model_size, models_dir=\"gpt2\")\n",
    "\n",
    "model = GPTModel(BASE_CONFIG)\n",
    "load_weights_into_gpt(model, params)\n",
    "model.eval();"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "id": "bc0985b2",
   "metadata": {},
   "outputs": [],
   "source": [
    "from utils import (\n",
    "    generate_text_simple,\n",
    "    text_to_token_ids,\n",
    "    token_ids_to_text\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "id": "2b3110bb",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Every effort moves you forward.\n",
      "\n",
      "The first step is to understand the importance of your work\n"
     ]
    }
   ],
   "source": [
    "text_1 = \"Every effort moves you\"\n",
    "\n",
    "token_ids = generate_text_simple(\n",
    "    model=model,\n",
    "    idx=text_to_token_ids(text_1, tokenizer),\n",
    "    max_new_tokens=15,\n",
    "    context_size=BASE_CONFIG[\"context_length\"]\n",
    ")\n",
    "\n",
    "print(token_ids_to_text(token_ids, tokenizer))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "id": "93610947",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Is the following text 'spam'? Answer with 'yes' or 'no': 'You are a winner you have been specially selected to receive $1000 cash or a $2000 award.'\n",
      "\n",
      "The following text 'spam'? Answer with 'yes' or 'no': 'You are a winner\n"
     ]
    }
   ],
   "source": [
    "text_2 = (\n",
    "    \"Is the following text 'spam'? Answer with 'yes' or 'no':\"\n",
    "    \" 'You are a winner you have been specially\"\n",
    "    \" selected to receive $1000 cash or a $2000 award.'\"\n",
    ")\n",
    "\n",
    "token_ids = generate_text_simple(\n",
    "    model=model,\n",
    "    idx=text_to_token_ids(text_2, tokenizer),\n",
    "    max_new_tokens=23,\n",
    "    context_size=BASE_CONFIG[\"context_length\"]\n",
    ")\n",
    "\n",
    "print(token_ids_to_text(token_ids, tokenizer))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "23954af8",
   "metadata": {},
   "source": [
    "## 添加分类头"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d77006ce",
   "metadata": {},
   "source": [
    "- 本节将修改预训练的大语言模型，为其进行分类微调做好准备；\n",
    "- 首先让我们审视模型架构；"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bf7ae794",
   "metadata": {},
   "source": [
    "![](./images/lm-head.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "id": "26b7b682",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "GPTModel(\n",
      "  (tok_emb): Embedding(50257, 768)\n",
      "  (pos_emb): Embedding(1024, 768)\n",
      "  (drop_emb): Dropout(p=0.0, inplace=False)\n",
      "  (trf_blocks): Sequential(\n",
      "    (0): TransformerBlock(\n",
      "      (att): MultiHeadAttention(\n",
      "        (W_query): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (W_key): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (W_value): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (out_proj): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (dropout): Dropout(p=0.0, inplace=False)\n",
      "      )\n",
      "      (ff): FeedForward(\n",
      "        (layers): Sequential(\n",
      "          (0): Linear(in_features=768, out_features=3072, bias=True)\n",
      "          (1): GELU()\n",
      "          (2): Linear(in_features=3072, out_features=768, bias=True)\n",
      "        )\n",
      "      )\n",
      "      (norm1): LayerNorm()\n",
      "      (norm2): LayerNorm()\n",
      "      (drop_resid): Dropout(p=0.0, inplace=False)\n",
      "    )\n",
      "    (1): TransformerBlock(\n",
      "      (att): MultiHeadAttention(\n",
      "        (W_query): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (W_key): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (W_value): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (out_proj): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (dropout): Dropout(p=0.0, inplace=False)\n",
      "      )\n",
      "      (ff): FeedForward(\n",
      "        (layers): Sequential(\n",
      "          (0): Linear(in_features=768, out_features=3072, bias=True)\n",
      "          (1): GELU()\n",
      "          (2): Linear(in_features=3072, out_features=768, bias=True)\n",
      "        )\n",
      "      )\n",
      "      (norm1): LayerNorm()\n",
      "      (norm2): LayerNorm()\n",
      "      (drop_resid): Dropout(p=0.0, inplace=False)\n",
      "    )\n",
      "    (2): TransformerBlock(\n",
      "      (att): MultiHeadAttention(\n",
      "        (W_query): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (W_key): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (W_value): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (out_proj): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (dropout): Dropout(p=0.0, inplace=False)\n",
      "      )\n",
      "      (ff): FeedForward(\n",
      "        (layers): Sequential(\n",
      "          (0): Linear(in_features=768, out_features=3072, bias=True)\n",
      "          (1): GELU()\n",
      "          (2): Linear(in_features=3072, out_features=768, bias=True)\n",
      "        )\n",
      "      )\n",
      "      (norm1): LayerNorm()\n",
      "      (norm2): LayerNorm()\n",
      "      (drop_resid): Dropout(p=0.0, inplace=False)\n",
      "    )\n",
      "    (3): TransformerBlock(\n",
      "      (att): MultiHeadAttention(\n",
      "        (W_query): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (W_key): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (W_value): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (out_proj): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (dropout): Dropout(p=0.0, inplace=False)\n",
      "      )\n",
      "      (ff): FeedForward(\n",
      "        (layers): Sequential(\n",
      "          (0): Linear(in_features=768, out_features=3072, bias=True)\n",
      "          (1): GELU()\n",
      "          (2): Linear(in_features=3072, out_features=768, bias=True)\n",
      "        )\n",
      "      )\n",
      "      (norm1): LayerNorm()\n",
      "      (norm2): LayerNorm()\n",
      "      (drop_resid): Dropout(p=0.0, inplace=False)\n",
      "    )\n",
      "    (4): TransformerBlock(\n",
      "      (att): MultiHeadAttention(\n",
      "        (W_query): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (W_key): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (W_value): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (out_proj): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (dropout): Dropout(p=0.0, inplace=False)\n",
      "      )\n",
      "      (ff): FeedForward(\n",
      "        (layers): Sequential(\n",
      "          (0): Linear(in_features=768, out_features=3072, bias=True)\n",
      "          (1): GELU()\n",
      "          (2): Linear(in_features=3072, out_features=768, bias=True)\n",
      "        )\n",
      "      )\n",
      "      (norm1): LayerNorm()\n",
      "      (norm2): LayerNorm()\n",
      "      (drop_resid): Dropout(p=0.0, inplace=False)\n",
      "    )\n",
      "    (5): TransformerBlock(\n",
      "      (att): MultiHeadAttention(\n",
      "        (W_query): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (W_key): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (W_value): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (out_proj): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (dropout): Dropout(p=0.0, inplace=False)\n",
      "      )\n",
      "      (ff): FeedForward(\n",
      "        (layers): Sequential(\n",
      "          (0): Linear(in_features=768, out_features=3072, bias=True)\n",
      "          (1): GELU()\n",
      "          (2): Linear(in_features=3072, out_features=768, bias=True)\n",
      "        )\n",
      "      )\n",
      "      (norm1): LayerNorm()\n",
      "      (norm2): LayerNorm()\n",
      "      (drop_resid): Dropout(p=0.0, inplace=False)\n",
      "    )\n",
      "    (6): TransformerBlock(\n",
      "      (att): MultiHeadAttention(\n",
      "        (W_query): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (W_key): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (W_value): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (out_proj): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (dropout): Dropout(p=0.0, inplace=False)\n",
      "      )\n",
      "      (ff): FeedForward(\n",
      "        (layers): Sequential(\n",
      "          (0): Linear(in_features=768, out_features=3072, bias=True)\n",
      "          (1): GELU()\n",
      "          (2): Linear(in_features=3072, out_features=768, bias=True)\n",
      "        )\n",
      "      )\n",
      "      (norm1): LayerNorm()\n",
      "      (norm2): LayerNorm()\n",
      "      (drop_resid): Dropout(p=0.0, inplace=False)\n",
      "    )\n",
      "    (7): TransformerBlock(\n",
      "      (att): MultiHeadAttention(\n",
      "        (W_query): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (W_key): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (W_value): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (out_proj): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (dropout): Dropout(p=0.0, inplace=False)\n",
      "      )\n",
      "      (ff): FeedForward(\n",
      "        (layers): Sequential(\n",
      "          (0): Linear(in_features=768, out_features=3072, bias=True)\n",
      "          (1): GELU()\n",
      "          (2): Linear(in_features=3072, out_features=768, bias=True)\n",
      "        )\n",
      "      )\n",
      "      (norm1): LayerNorm()\n",
      "      (norm2): LayerNorm()\n",
      "      (drop_resid): Dropout(p=0.0, inplace=False)\n",
      "    )\n",
      "    (8): TransformerBlock(\n",
      "      (att): MultiHeadAttention(\n",
      "        (W_query): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (W_key): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (W_value): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (out_proj): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (dropout): Dropout(p=0.0, inplace=False)\n",
      "      )\n",
      "      (ff): FeedForward(\n",
      "        (layers): Sequential(\n",
      "          (0): Linear(in_features=768, out_features=3072, bias=True)\n",
      "          (1): GELU()\n",
      "          (2): Linear(in_features=3072, out_features=768, bias=True)\n",
      "        )\n",
      "      )\n",
      "      (norm1): LayerNorm()\n",
      "      (norm2): LayerNorm()\n",
      "      (drop_resid): Dropout(p=0.0, inplace=False)\n",
      "    )\n",
      "    (9): TransformerBlock(\n",
      "      (att): MultiHeadAttention(\n",
      "        (W_query): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (W_key): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (W_value): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (out_proj): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (dropout): Dropout(p=0.0, inplace=False)\n",
      "      )\n",
      "      (ff): FeedForward(\n",
      "        (layers): Sequential(\n",
      "          (0): Linear(in_features=768, out_features=3072, bias=True)\n",
      "          (1): GELU()\n",
      "          (2): Linear(in_features=3072, out_features=768, bias=True)\n",
      "        )\n",
      "      )\n",
      "      (norm1): LayerNorm()\n",
      "      (norm2): LayerNorm()\n",
      "      (drop_resid): Dropout(p=0.0, inplace=False)\n",
      "    )\n",
      "    (10): TransformerBlock(\n",
      "      (att): MultiHeadAttention(\n",
      "        (W_query): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (W_key): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (W_value): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (out_proj): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (dropout): Dropout(p=0.0, inplace=False)\n",
      "      )\n",
      "      (ff): FeedForward(\n",
      "        (layers): Sequential(\n",
      "          (0): Linear(in_features=768, out_features=3072, bias=True)\n",
      "          (1): GELU()\n",
      "          (2): Linear(in_features=3072, out_features=768, bias=True)\n",
      "        )\n",
      "      )\n",
      "      (norm1): LayerNorm()\n",
      "      (norm2): LayerNorm()\n",
      "      (drop_resid): Dropout(p=0.0, inplace=False)\n",
      "    )\n",
      "    (11): TransformerBlock(\n",
      "      (att): MultiHeadAttention(\n",
      "        (W_query): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (W_key): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (W_value): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (out_proj): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (dropout): Dropout(p=0.0, inplace=False)\n",
      "      )\n",
      "      (ff): FeedForward(\n",
      "        (layers): Sequential(\n",
      "          (0): Linear(in_features=768, out_features=3072, bias=True)\n",
      "          (1): GELU()\n",
      "          (2): Linear(in_features=3072, out_features=768, bias=True)\n",
      "        )\n",
      "      )\n",
      "      (norm1): LayerNorm()\n",
      "      (norm2): LayerNorm()\n",
      "      (drop_resid): Dropout(p=0.0, inplace=False)\n",
      "    )\n",
      "  )\n",
      "  (final_norm): LayerNorm()\n",
      "  (out_head): Linear(in_features=768, out_features=50257, bias=False)\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "print(model)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f7f7f79b",
   "metadata": {},
   "source": [
    "- 如上所示，我们可以清晰看到第四章所实现的模型架构；\n",
    "- 我们的目标是替换并微调输出层；\n",
    "- 为实现此目标，我们首先冻结模型，即使所有层变为不可训练状态；"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "id": "3410446c",
   "metadata": {},
   "outputs": [],
   "source": [
    "for param in model.parameters():\n",
    "    param.requires_grad = False"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ee84dcd4",
   "metadata": {},
   "source": [
    "- 随后，我们替换输出层（model.out_head），该层原本将输入映射至50,257维（对应词汇表大小）；\n",
    "- 由于本次微调针对二分类任务（预测\"垃圾邮件\"和\"非垃圾邮件\"两个类别），我们可以按下文所示替换输出层，该层默认处于可训练状态；\n",
    "- 请注意，我们使用BASE_CONFIG[\"emb_dim\"]（在\"gpt2-small (124M)\"模型中该值为768）以保持后续代码的通用性；\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "id": "bd94a934",
   "metadata": {},
   "outputs": [],
   "source": [
    "torch.manual_seed(123)\n",
    "\n",
    "num_classes = 2\n",
    "model.out_head = torch.nn.Linear(in_features=BASE_CONFIG[\"emb_dim\"], out_features=num_classes)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b2bfce1d",
   "metadata": {},
   "source": [
    "- 从技术角度而言，仅训练输出层已足够；\n",
    "- 实验表明微调更多层能显著提升性能；\n",
    "- 因此，我们还将最后一个Transformer模块及其连接输出层的最终LayerNorm模块设为可训练状态；\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "73640c1b",
   "metadata": {},
   "source": [
    "![](./images/trainable.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "id": "adb7ca7f",
   "metadata": {},
   "outputs": [],
   "source": [
    "for param in model.trf_blocks[-1].parameters():\n",
    "    param.requires_grad = True\n",
    "\n",
    "for param in model.final_norm.parameters():\n",
    "    param.requires_grad = True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "id": "698e1f74",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Inputs: tensor([[5211,  345,  423,  640]])\n",
      "Inputs dimensions: torch.Size([1, 4])\n"
     ]
    }
   ],
   "source": [
    "inputs = tokenizer.encode(\"Do you have time\")\n",
    "inputs = torch.tensor(inputs).unsqueeze(0)\n",
    "print(\"Inputs:\", inputs)\n",
    "print(\"Inputs dimensions:\", inputs.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "id": "782316ef",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Outputs:\n",
      " tensor([[[-1.5854,  0.9904],\n",
      "         [-3.7235,  7.4548],\n",
      "         [-2.2661,  6.6049],\n",
      "         [-3.5983,  3.9902]]])\n",
      "Outputs dimensions: torch.Size([1, 4, 2])\n"
     ]
    }
   ],
   "source": [
    "with torch.no_grad():\n",
    "    outputs = model(inputs)\n",
    "\n",
    "print(\"Outputs:\\n\", outputs)\n",
    "print(\"Outputs dimensions:\", outputs.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5203a53d",
   "metadata": {},
   "source": [
    "### 计算分类损失与准确率"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5a3f08b6",
   "metadata": {},
   "source": [
    "在阐述损失计算之前，我们先简要了解模型输出如何转换为类别标签：模型最后一层输出每个类别的原始预测值（logits）；通过softmax函数将这些logits转换为概率分布；最终预测类别由最高概率对应的标签决定（通过argmax操作实现）"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9cc78beb",
   "metadata": {},
   "source": [
    "![](./images/class-argmax.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "id": "0f1bb005",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Last output token: tensor([[-3.5983,  3.9902]])\n"
     ]
    }
   ],
   "source": [
    "print(\"Last output token:\", outputs[:, -1, :])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "id": "77b8633e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Class label: 1\n"
     ]
    }
   ],
   "source": [
    "probas = torch.softmax(outputs[:, -1, :], dim=-1)\n",
    "label = torch.argmax(probas)\n",
    "print(\"Class label:\", label.item())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ad9d80dc",
   "metadata": {},
   "source": [
    "- 我们可以应用这一概念计算所谓的​​分类准确率​​，即统计给定数据集中正确预测的百分比；\n",
    "- 为计算分类准确率，可将前述基于argmax的预测代码应用于数据集中的所有样本，并通过以下方式计算正确预测的比例；"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "id": "cc97d750",
   "metadata": {},
   "outputs": [],
   "source": [
    "def calc_accuracy_loader(data_loader, model, device, num_batches=None):\n",
    "    model.eval()\n",
    "    correct_predictions, num_examples = 0, 0\n",
    "\n",
    "    if num_batches is None:\n",
    "        num_batches = len(data_loader)\n",
    "    else:\n",
    "        num_batches = min(num_batches, len(data_loader))\n",
    "    for i, (input_batch, target_batch) in enumerate(data_loader):\n",
    "        if i < num_batches:\n",
    "            input_batch, target_batch = input_batch.to(device), target_batch.to(device)\n",
    "\n",
    "            with torch.no_grad():\n",
    "                logits = model(input_batch)[:, -1, :]  # Logits of last output token\n",
    "            predicted_labels = torch.argmax(logits, dim=-1)\n",
    "\n",
    "            num_examples += predicted_labels.shape[0]\n",
    "            correct_predictions += (predicted_labels == target_batch).sum().item()\n",
    "        else:\n",
    "            break\n",
    "    return correct_predictions / num_examples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "id": "e2de1e2c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training accuracy: 46.25%\n",
      "Validation accuracy: 45.00%\n",
      "Test accuracy: 48.75%\n"
     ]
    }
   ],
   "source": [
    "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
    "\n",
    "model.to(device)\n",
    "\n",
    "torch.manual_seed(123)\n",
    "\n",
    "train_accuracy = calc_accuracy_loader(train_loader, model, device, num_batches=10)\n",
    "val_accuracy = calc_accuracy_loader(val_loader, model, device, num_batches=10)\n",
    "test_accuracy = calc_accuracy_loader(test_loader, model, device, num_batches=10)\n",
    "\n",
    "print(f\"Training accuracy: {train_accuracy*100:.2f}%\")\n",
    "print(f\"Validation accuracy: {val_accuracy*100:.2f}%\")\n",
    "print(f\"Test accuracy: {test_accuracy*100:.2f}%\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "885bd09b",
   "metadata": {},
   "source": [
    "- 在开始微调（训练）之前，我们首先需要定义训练过程中待优化的损失函数；\n",
    "- 我们的目标是最大化模型的垃圾邮件分类准确率；然而，分类准确率本身是不可微的函数；\n",
    "- 因此，我们转而最小化交叉熵损失，将其作为最大化分类准确率的代理目标；\n",
    "- 此处的calc_loss_batch函数与第5章相同，区别在于我们仅需优化最后一个词元的输出model(input_batch)[:, -1, :]，而非所有词元的输出model(input_batch)；\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "id": "7657591f",
   "metadata": {},
   "outputs": [],
   "source": [
    "def calc_loss_batch(input_batch, target_batch, model, device):\n",
    "    input_batch, target_batch = input_batch.to(device), target_batch.to(device)\n",
    "    logits = model(input_batch)[:, -1, :]  # Logits of last output token\n",
    "    loss = torch.nn.functional.cross_entropy(logits, target_batch)\n",
    "    return loss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "id": "67932ee2",
   "metadata": {},
   "outputs": [],
   "source": [
    "def calc_loss_loader(data_loader, model, device, num_batches=None):\n",
    "    total_loss = 0.\n",
    "    if len(data_loader) == 0:\n",
    "        return float(\"nan\")\n",
    "    elif num_batches is None:\n",
    "        num_batches = len(data_loader)\n",
    "    else:\n",
    "        num_batches = min(num_batches, len(data_loader))\n",
    "    for i, (input_batch, target_batch) in enumerate(data_loader):\n",
    "        if i < num_batches:\n",
    "            loss = calc_loss_batch(input_batch, target_batch, model, device)\n",
    "            total_loss += loss.item()\n",
    "        else:\n",
    "            break\n",
    "    return total_loss / num_batches"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "id": "2ab60372",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training loss: 2.453\n",
      "Validation loss: 2.583\n",
      "Test loss: 2.322\n"
     ]
    }
   ],
   "source": [
    "with torch.no_grad():\n",
    "    train_loss = calc_loss_loader(train_loader, model, device, num_batches=5)\n",
    "    val_loss = calc_loss_loader(val_loader, model, device, num_batches=5)\n",
    "    test_loss = calc_loss_loader(test_loader, model, device, num_batches=5)\n",
    "\n",
    "print(f\"Training loss: {train_loss:.3f}\")\n",
    "print(f\"Validation loss: {val_loss:.3f}\")\n",
    "print(f\"Test loss: {test_loss:.3f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "49e94b66",
   "metadata": {},
   "source": [
    "## 在有监督数据上微调模型"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3a3710e9",
   "metadata": {},
   "source": [
    "- 本节将定义并使用训练函数来提升模型的分类准确率；\n",
    "- 下文中的train_classifier_simple函数与第5章用于预训练模型的train_model_simple函数几乎相同；\n",
    "- 仅存在两点区别：\n",
    "    - 当前记录已观测的训练样本数（examples_seen）而非已观测的词元数；\n",
    "    - 在每个训练轮次后计算准确率，而非在每个轮次后打印示例文本；\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "id": "fbe73bcc",
   "metadata": {},
   "outputs": [],
   "source": [
    "def evaluate_model(model, train_loader, val_loader, device, eval_iter):\n",
    "    model.eval()\n",
    "    with torch.no_grad():\n",
    "        train_loss = calc_loss_loader(train_loader, model, device, num_batches=eval_iter)\n",
    "        val_loss = calc_loss_loader(val_loader, model, device, num_batches=eval_iter)\n",
    "    model.train()\n",
    "    return train_loss, val_loss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "id": "ac561866",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Overall the same as `train_model_simple` in chapter 5\n",
    "def train_classifier_simple(model, train_loader, val_loader, optimizer, device, num_epochs,\n",
    "                            eval_freq, eval_iter):\n",
    "    # Initialize lists to track losses and examples seen\n",
    "    train_losses, val_losses, train_accs, val_accs = [], [], [], []\n",
    "    examples_seen, global_step = 0, -1\n",
    "\n",
    "    # Main training loop\n",
    "    for epoch in range(num_epochs):\n",
    "        model.train()  # Set model to training mode\n",
    "\n",
    "        for input_batch, target_batch in train_loader:\n",
    "            optimizer.zero_grad() # Reset loss gradients from previous batch iteration\n",
    "            loss = calc_loss_batch(input_batch, target_batch, model, device)\n",
    "            loss.backward() # Calculate loss gradients\n",
    "            optimizer.step() # Update model weights using loss gradients\n",
    "            examples_seen += input_batch.shape[0] # New: track examples instead of tokens\n",
    "            global_step += 1\n",
    "\n",
    "            # Optional evaluation step\n",
    "            if global_step % eval_freq == 0:\n",
    "                train_loss, val_loss = evaluate_model(\n",
    "                    model, train_loader, val_loader, device, eval_iter)\n",
    "                train_losses.append(train_loss)\n",
    "                val_losses.append(val_loss)\n",
    "                print(f\"Ep {epoch+1} (Step {global_step:06d}): \"\n",
    "                      f\"Train loss {train_loss:.3f}, Val loss {val_loss:.3f}\")\n",
    "\n",
    "        # Calculate accuracy after each epoch\n",
    "        train_accuracy = calc_accuracy_loader(train_loader, model, device, num_batches=eval_iter)\n",
    "        val_accuracy = calc_accuracy_loader(val_loader, model, device, num_batches=eval_iter)\n",
    "        print(f\"Training accuracy: {train_accuracy*100:.2f}% | \", end=\"\")\n",
    "        print(f\"Validation accuracy: {val_accuracy*100:.2f}%\")\n",
    "        train_accs.append(train_accuracy)\n",
    "        val_accs.append(val_accuracy)\n",
    "\n",
    "    return train_losses, val_losses, train_accs, val_accs, examples_seen"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "id": "6baa4ece",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Ep 1 (Step 000000): Train loss 2.153, Val loss 2.392\n",
      "Ep 1 (Step 000050): Train loss 0.617, Val loss 0.637\n",
      "Ep 1 (Step 000100): Train loss 0.523, Val loss 0.557\n",
      "Training accuracy: 70.00% | Validation accuracy: 72.50%\n",
      "Ep 2 (Step 000150): Train loss 0.561, Val loss 0.489\n",
      "Ep 2 (Step 000200): Train loss 0.419, Val loss 0.397\n",
      "Ep 2 (Step 000250): Train loss 0.409, Val loss 0.353\n",
      "Training accuracy: 82.50% | Validation accuracy: 85.00%\n",
      "Ep 3 (Step 000300): Train loss 0.333, Val loss 0.320\n",
      "Ep 3 (Step 000350): Train loss 0.340, Val loss 0.306\n",
      "Training accuracy: 90.00% | Validation accuracy: 90.00%\n",
      "Ep 4 (Step 000400): Train loss 0.136, Val loss 0.200\n",
      "Ep 4 (Step 000450): Train loss 0.153, Val loss 0.132\n",
      "Ep 4 (Step 000500): Train loss 0.222, Val loss 0.137\n",
      "Training accuracy: 100.00% | Validation accuracy: 97.50%\n",
      "Ep 5 (Step 000550): Train loss 0.207, Val loss 0.143\n",
      "Ep 5 (Step 000600): Train loss 0.083, Val loss 0.074\n",
      "Training accuracy: 100.00% | Validation accuracy: 97.50%\n",
      "Training completed in 22.56 minutes.\n"
     ]
    }
   ],
   "source": [
    "import time\n",
    "\n",
    "start_time = time.time()\n",
    "\n",
    "torch.manual_seed(123)\n",
    "\n",
    "optimizer = torch.optim.AdamW(model.parameters(), lr=5e-5, weight_decay=0.1)\n",
    "\n",
    "num_epochs = 5\n",
    "train_losses, val_losses, train_accs, val_accs, examples_seen = train_classifier_simple(\n",
    "    model, train_loader, val_loader, optimizer, device,\n",
    "    num_epochs=num_epochs, eval_freq=50, eval_iter=5,\n",
    ")\n",
    "\n",
    "end_time = time.time()\n",
    "execution_time_minutes = (end_time - start_time) / 60\n",
    "print(f\"Training completed in {execution_time_minutes:.2f} minutes.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "id": "94bcd495",
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "def plot_values(epochs_seen, examples_seen, train_values, val_values, label=\"loss\"):\n",
    "    fig, ax1 = plt.subplots(figsize=(5, 3))\n",
    "\n",
    "    # Plot training and validation loss against epochs\n",
    "    ax1.plot(epochs_seen, train_values, label=f\"Training {label}\")\n",
    "    ax1.plot(epochs_seen, val_values, linestyle=\"-.\", label=f\"Validation {label}\")\n",
    "    ax1.set_xlabel(\"Epochs\")\n",
    "    ax1.set_ylabel(label.capitalize())\n",
    "    ax1.legend()\n",
    "\n",
    "    # Create a second x-axis for examples seen\n",
    "    ax2 = ax1.twiny()  # Create a second x-axis that shares the same y-axis\n",
    "    ax2.plot(examples_seen, train_values, alpha=0)  # Invisible plot for aligning ticks\n",
    "    ax2.set_xlabel(\"Examples seen\")\n",
    "\n",
    "    fig.tight_layout()  # Adjust layout to make room\n",
    "    plt.savefig(f\"{label}-plot.pdf\")\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "id": "4c03e251",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAeoAAAEiCAYAAAA21pHjAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjMsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvZiW1igAAAAlwSFlzAAAPYQAAD2EBqD+naQAAV4FJREFUeJzt3XlcVPX++PHXzMAM+74j4ga4gru5U5JLZdnq1+stLctbYWVmi7dSs1/RYjcrzcpucutWlpbWLZcQ931FwQV3QGVzYRUGmDm/PwZGR3EBgRnw/Xw8zoM5n/M557znE/nmfM7nnI9KURQFIYQQQtgktbUDEEIIIcTVSaIWQgghbJgkaiGEEMKGSaIWQgghbJgkaiGEEMKGSaIWQgghbJgkaiGEEMKGSaIWQgghbJgkaiGEEMKGSaIWQtyQ6OhoJk6caO0whLjlSKIWooGMHTsWlUp1xTJ06FBrhyaEsGF21g5AiFvJ0KFDmT9/vkWZTqezUjRCiMZArqiFaEA6nY6AgACLxdPTE4A1a9ag1WpZv369uf4HH3yAn58f2dnZACxfvpx+/frh4eGBt7c399xzD0ePHjXXP3HiBCqVip9//pn+/fvj6OhIjx49OHToENu3b6d79+64uLgwbNgwcnNzzfuNHTuWESNG8NZbb+Hr64ubmxtPP/00ZWVlV/0uer2eyZMnExwcjLOzM7169WLNmjXm7WlpaQwfPhxPT0+cnZ3p0KEDS5cuverxPv/8c8LCwnBwcMDf35+HHnrIvM1oNBIXF0fLli1xdHQkKiqKRYsWWeyfkpLCsGHDcHFxwd/fn0cffZQzZ86Yt0dHR/P888/zyiuv4OXlRUBAANOnT79qPELYCknUQtiIqnvAjz76KPn5+ezevZs333yTr7/+Gn9/fwCKi4uZNGkSO3bsIDExEbVazf3334/RaLQ41rRp03jjjTfYtWsXdnZ2/O1vf+OVV17hk08+Yf369Rw5coSpU6da7JOYmMiBAwdYs2YNP/74I7/++itvvfXWVeOdMGECmzdvZsGCBezdu5eHH36YoUOHcvjwYQBiY2PR6/WsW7eO5ORk3n//fVxcXKo91o4dO3j++eeZMWMGqampLF++nAEDBpi3x8XF8e233/LFF1+wb98+XnzxRf7+97+zdu1aAPLy8rjjjjvo0qULO3bsYPny5WRnZ/PII49YnOc///kPzs7ObN26lQ8++IAZM2aQkJBwg/+FhLASRQjRIMaMGaNoNBrF2dnZYnnnnXfMdfR6vdK5c2flkUceUdq3b6889dRT1zxmbm6uAijJycmKoijK8ePHFUD5+uuvzXV+/PFHBVASExPNZXFxcUpERIRFbF5eXkpxcbG5bO7cuYqLi4tiMBgURVGUgQMHKi+88IKiKIqSlpamaDQa5dSpUxbxDBo0SJkyZYqiKIrSqVMnZfr06TfUNr/88ovi5uamFBQUXLGttLRUcXJyUjZt2mRRPm7cOGXUqFGKoijK22+/rQwePNhie0ZGhgIoqamp5vj79etnUadHjx7Kq6++ekMxCmEtco9aiAZ0++23M3fuXIsyLy8v82etVsv3339PZGQkoaGhfPzxxxZ1Dx8+zNSpU9m6dStnzpwxX0mnp6fTsWNHc73IyEjz56qr8U6dOlmU5eTkWBw7KioKJycn83rv3r0pKioiIyOD0NBQi7rJyckYDAbCw8MtyvV6Pd7e3gA8//zzPPPMM/z111/ExMTw4IMPWsR1qTvvvJPQ0FBatWrF0KFDGTp0KPfffz9OTk4cOXKECxcucOedd1rsU1ZWRpcuXQDYs2cPq1evrvaK/ejRo+Y4Lz9/YGDgFe0ghK2RRC1EA3J2dqZNmzbXrLNp0yYAzp07x7lz53B2djZvGz58OKGhocybN4+goCCMRiMdO3a84l6yvb29+bNKpaq27PLu8pooKipCo9Gwc+dONBqNxbaqZPnkk08yZMgQ/vzzT/766y/i4uL46KOPeO655644nqurK7t27WLNmjX89ddfTJ06lenTp7N9+3aKiooA+PPPPwkODrbYr2ogXlFREcOHD+f999+/4tiBgYHmz5e2Adx8OwjRECRRC2FDjh49yosvvsi8efP46aefGDNmDCtXrkStVnP27FlSU1OZN28e/fv3B2DDhg11du49e/ZQUlKCo6MjAFu2bMHFxYWQkJAr6nbp0gWDwUBOTo45luqEhITw9NNP8/TTTzNlyhTmzZtXbaIGsLOzIyYmhpiYGKZNm4aHhwerVq3izjvvRKfTkZ6ezsCBA6vdt2vXrvzyyy+0aNECOzv5Z000LfIbLUQD0uv1ZGVlWZTZ2dnh4+ODwWDg73//O0OGDOHxxx9n6NChdOrUiY8++oiXX34ZT09PvL29+eqrrwgMDCQ9PZ3XXnutzmIrKytj3LhxvPHGG5w4cYJp06YxYcIE1Oorx5yGh4czevRoHnvsMT766CO6dOlCbm4uiYmJREZGcvfddzNx4kSGDRtGeHg458+fZ/Xq1bRr167ac//xxx8cO3aMAQMG4OnpydKlSzEajURERODq6srkyZN58cUXMRqN9OvXj/z8fDZu3IibmxtjxowhNjaWefPmMWrUKPOo7iNHjrBgwQK+/vrrK676hWhMJFEL0YCWL19u0RULEBERwcGDB3nnnXdIS0vjjz/+AExdtl999RWjRo1i8ODBREVFsWDBAp5//nk6duxIREQEn376KdHR0XUS26BBgwgLC2PAgAHo9XpGjRp1zceX5s+fz//7f/+Pl156iVOnTuHj48Ntt93GPffcA4DBYCA2NpaTJ0/i5ubG0KFDr7jnXsXDw4Nff/2V6dOnU1paSlhYGD/++CMdOnQA4O2338bX15e4uDiOHTuGh4cHXbt25Z///CcAQUFBbNy4kVdffZXBgwej1+sJDQ1l6NCh1f6hIURjolIURbF2EEII6xo7dix5eXksWbLE2qEIIS4jf2oKIYQQNkwStRBCCGHDpOtbCCGEsGFyRS2EEELYMEnUQgghhA2TRC2EEELYMEnUN2HOnDm0aNECBwcHevXqxbZt26wdUr1Zt24dw4cPJygoCJVKdcVjPIqiMHXqVAIDA3F0dCQmJsY8i1KVc+fOMXr0aNzc3PDw8GDcuHHm10NW2bt3L/3798fBwYGQkBA++OCD+v5qdSIuLo4ePXrg6uqKn58fI0aMIDU11aJOaWkpsbGxeHt74+LiwoMPPmievrJKeno6d999N05OTvj5+fHyyy9TUVFhUWfNmjV07doVnU5HmzZtiI+Pr++vVyfmzp1LZGQkbm5uuLm50bt3b5YtW2befqu3T3Xee+89VCoVEydONJdJO8H06dNRqVQWS9u2bc3bm1wbWXVKkEZswYIFilarVb755htl3759ylNPPaV4eHgo2dnZ1g6tXixdulR5/fXXlV9//VUBlMWLF1tsf++99xR3d3dlyZIlyp49e5R7771XadmypVJSUmKuM3ToUCUqKkrZsmWLsn79eqVNmzbm2Y8URVHy8/MVf39/ZfTo0UpKSory448/Ko6OjsqXX37ZUF+z1oYMGaLMnz9fSUlJUZKSkpS77rpLad68uVJUVGSu8/TTTyshISFKYmKismPHDuW2225T+vTpY95eUVGhdOzYUYmJiVF2796tLF26VPHx8THPRqUoinLs2DHFyclJmTRpkrJ//37ls88+UzQajbJ8+fIG/b618fvvvyt//vmncujQISU1NVX55z//qdjb2yspKSmKokj7XG7btm1KixYtlMjISPOsZYoi7aQoijJt2jSlQ4cOSmZmpnnJzc01b29qbSSJupZ69uypxMbGmtcNBoMSFBSkxMXFWTGqhnF5ojYajUpAQIDy4Ycfmsvy8vIUnU6n/Pjjj4qiKMr+/fsVQNm+fbu5zrJlyxSVSmWeKvHzzz9XPD09Fb1eb67z6quvWkzH2Fjk5OQogLJ27VpFUUztYW9vryxcuNBc58CBAwqgbN68WVEU0x9DarVaycrKMteZO3eu4ubmZm6TV155RenQoYPFuUaOHKkMGTKkvr9SvfD09FS+/vpraZ/LFBYWKmFhYUpCQoLF9KLSTibTpk1ToqKiqt3WFNtIur5roaysjJ07dxITE2MuU6vVxMTEsHnzZitGZh3Hjx8nKyvLoj3c3d3p1auXuT02b96Mh4cH3bt3N9eJiYlBrVazdetWc50BAwag1WrNdYYMGUJqairnz59voG9TN/Lz84GLU1ju3LmT8vJyizZq27YtzZs3t2ijTp06maelBNP3LygoYN++feY6lx6jqk5j+70zGAwsWLCA4uJievfuLe1zmdjYWO6+++4rvou000WHDx8mKCiIVq1aMXr0aNLT04Gm2UaSqGvhzJkzGAwGi//IYJrj9/IJF24FVd/5Wu2RlZWFn5+fxXY7Ozu8vLws6lR3jEvP0RgYjUYmTpxI3759zXNEZ2VlodVq8fDwsKh7eRtd7/tfrU5BQQElJSX18XXqVHJyMi4uLuh0Op5++mkWL15M+/btpX0usWDBAnbt2kVcXNwV26SdTHr16kV8fDzLly9n7ty5HD9+nP79+1NYWNgk20gm5RCijsXGxpKSklKnU1A2FRERESQlJZGfn8+iRYsYM2YMa9eutXZYNiMjI4MXXniBhIQEHBwcrB2OzRo2bJj5c2RkJL169SI0NJSff/7ZPE1rUyJX1LXg4+ODRqO5YhRhdnY2AQEBVorKeqq+87XaIyAggJycHIvtFRUVnDt3zqJOdce49By2bsKECfzxxx+sXr2aZs2amcsDAgIoKysjLy/Pov7lbXS973+1Om5ubo3iHyitVkubNm3o1q0bcXFxREVF8cknn0j7VNq5cyc5OTl07doVOzs77OzsWLt2LZ9++il2dnb4+/tLO1XDw8OD8PBwjhw50iR/lyRR14JWq6Vbt24kJiaay4xGI4mJifTu3duKkVlHy5YtCQgIsGiPgoICtm7dam6P3r17k5eXx86dO811Vq1ahdFopFevXuY669ato7y83FwnISGBiIgIPD09G+jb1I6iKEyYMIHFixezatUqWrZsabG9W7du2NvbW7RRamoq6enpFm2UnJxs8QdNQkICbm5utG/f3lzn0mNU1Wmsv3dGoxG9Xi/tU2nQoEEkJyeTlJRkXrp3787o0aPNn6WdrlRUVMTRo0cJDAxsmr9LDT58rYlYsGCBotPplPj4eGX//v3K+PHjFQ8PD4tRhE1JYWGhsnv3bmX37t0KoPzrX/9Sdu/eraSlpSmKYno8y8PDQ/ntt9+UvXv3Kvfdd1+1j2d16dJF2bp1q7JhwwYlLCzM4vGsvLw8xd/fX3n00UeVlJQUZcGCBYqTk1OjeDzrmWeeUdzd3ZU1a9ZYPDJy4cIFc52nn35aad68ubJq1Splx44dSu/evZXevXubt1c9MjJ48GAlKSlJWb58ueLr61vtIyMvv/yycuDAAWXOnDmN5rGa1157TVm7dq1y/PhxZe/evcprr72mqFQq5a+//lIURdrnai4d9a0o0k6KoigvvfSSsmbNGuX48ePKxo0blZiYGMXHx0fJyclRFKXptZEk6pvw2WefKc2bN1e0Wq3Ss2dPZcuWLdYOqd6sXr1aAa5YxowZoyiK6RGtN998U/H391d0Op0yaNAgJTU11eIYZ8+eVUaNGqW4uLgobm5uyuOPP64UFhZa1NmzZ4/Sr18/RafTKcHBwcp7773XUF/xplTXNoAyf/58c52SkhLl2WefVTw9PRUnJyfl/vvvVzIzMy2Oc+LECWXYsGGKo6Oj4uPjo7z00ktKeXm5RZ3Vq1crnTt3VrRardKqVSuLc9iyJ554QgkNDVW0Wq3i6+urDBo0yJykFUXa52ouT9TSTqbHpAIDAxWtVqsEBwcrI0eOVI4cOWLe3tTaSGbPEkIIIWyY3KMWQgghbJgkaiGEEMKGSaIWQgghbJgkaiGEEMKGSaIWQgghbJgkaiGEEMKGSaK+CXq9nunTp6PX660dik2Tdro+aaPrkza6Pmmj62uMbWTV56jj4uL49ddfOXjwII6OjvTp04f333+fiIiIq+4THx/P448/blGm0+koLS2t73CvUFBQgLu7O/n5+bi5uTX4+RsLaafrkza6Pmmj65M2ur7G2EZWvaJeu3YtsbGxbNmyhYSEBMrLyxk8eDDFxcXX3M/NzY3MzEzzkpaW1kARCyGEEA3LqtNcLl++3GI9Pj4ePz8/du7cyYABA666n0qlajSzKQkhhBA3w6bmo87PzwfAy8vrmvWKiooIDQ3FaDTStWtX3n33XTp06HBD56ioqGD37t34+/ujVt9ch0JhYSEAp06doqCg4KaO1ZRJO12ftNH1SRtdn7TR9dlKGxmNRrKzs+nSpQt2dtdOxTbzrm+j0ci9995LXl4eGzZsuGq9zZs3c/jwYSIjI8nPz2fmzJmsW7eOffv2Wcz/W0Wv11sMGti5cyd33HFHvXwHIYQQoia2bdtGjx49rlnHZhL1M888w7Jly9iwYUO1CfdqysvLadeuHaNGjeLtt9++Yvv06dN56623rijftm0bgYGBNxWzEEIIURuZmZn07NmTtLQ0mjdvfs26NpGoJ0yYwG+//ca6deto2bJljfd/+OGHsbOz48cff7xi2+VX1KdOnaJ9+/ZkZGTU6A8CIYQQoq6cPHmSkJCQG8pFVh31rSgKEyZMYPHixaxatapWSdpgMJCcnHzVq2OdToebm5t5cXV1vdmwhRBCiAZj1cFksbGx/PDDD/z222+4urqSlZUFgLu7O46OjgA89thjBAcHExcXB8CMGTO47bbbaNOmDXl5eXz44YekpaXx5JNPWu17CCGEEPXFqol67ty5AERHR1uUz58/n7FjxwKQnp5uMTr7/PnzPPXUU2RlZeHp6Um3bt3YtGkT7du3b6iwhRBCiAZjE/eoG1JN7gsIIW49BoOB8vJya4chGjl7e3s0Gs1Vt9ckF9nUc9RCCGEtiqKQlZVFXl6etUMRTYSHhwcBAQGoVKqbOo4k6ptRkgfpW8C9GQR0tHY0QoibUJWk/fz8cHJyuul/XMWtS1EULly4QE5ODsBNPwosifpmrPp/sH0e9Hoahr1v7WiEELVkMBjMSdrb29va4YgmoGpAdE5ODn5+ftfsBr8emebyZrToa/p5YqN14xBC3JSqe9JOTk5WjkQ0JVW/Tzc75kES9c0IrUzU2Slw4Zx1YxFC3DTp7hZ1qa5+nyRR3wwXP/AJBxRI32ztaIQQQjRBkqhvVot+pp/S/S2EaCJatGjBrFmzbrj+mjVrUKlU9T5iPj4+Hg8Pj3o9hy2SRH2zqrq/T6y3bhxCiFuOSqW65jJ9+vRaHXf79u2MHz/+huv36dOHzMxM3N3da3U+cW0y6vtmVV1RZyWbHtdy9LBmNEKIW0hmZqb5808//cTUqVNJTU01l7m4uJg/K4qCwWC47tzHAL6+vjWKQ6vVEhAQUKN9xI2TK+qb5RoA3m0w3afeYu1ohBC3kICAAPPi7u6OSqUyrx88eBBXV1eWLVtGt27d0Ol0bNiwgaNHj3Lffffh7++Pi4sLPXr0YOXKlRbHvbzrW6VS8fXXX3P//ffj5OREWFgYv//+u3n75V3fVV3UK1asoF27dri4uDB06FCLPywqKip4/vnn8fDwwNvbm1dffZUxY8YwYsSIGrXB3Llzad26NVqtloiICL777jvzNkVRmD59Os2bN0en0xEUFMTzzz9v3v75558TFhaGg4MD/v7+PPTQQzU6d0ORRF0XpPtbiCZHURQulFVYZanLNzu/9tprvPfeexw4cIDIyEiKioq46667SExMZPfu3QwdOpThw4eTnp5+zeO89dZbPPLII+zdu5e77rqL0aNHc+7c1Z92uXDhAjNnzuS7775j3bp1pKenM3nyZPP2999/n++//5758+ezceNGCgoKWLJkSY2+2+LFi3nhhRd46aWXSElJ4R//+AePP/44q1evBuCXX37h448/5ssvv+Tw4cMsWbKETp06AbBjxw6ef/55ZsyYQWpqKsuXL2fAgAE1On9Dka7vutCiH+z6D6TJgDIhmoqScgPtp66wyrn3zxiCk7Zu/nmeMWMGd955p3ndy8uLqKgo8/rbb7/N4sWL+f3335kwYcJVjzN27FhGjRoFwLvvvsunn37Ktm3bGDp0aLX1y8vL+eKLL2jdujUAEyZMYMaMGebtn332GVOmTOH+++8HYPbs2SxdurRG323mzJmMHTuWZ599FoBJkyaxZcsWZs6cye233056ejoBAQHExMRgb29P8+bN6dmzJ2Ca8MnZ2Zl77rkHV1dXQkND6dKlS43O31DkirouVF1RZ+6B0nzrxiKEEJfo3r27xXpRURGTJ0+mXbt2eHh44OLiwoEDB657RR0ZGWn+7OzsjJubm/kVmdVxcnIyJ2kwvUazqn5+fj7Z2dnmpAmg0Wjo1q1bjb7bgQMH6Nu3r0VZ3759OXDgAAAPP/wwJSUltGrViqeeeorFixdTUVEBwJ133kloaCitWrXi0Ucf5fvvv+fChQs1On9DkSvquuAeDJ4t4fxxSN8K4YOtHZEQ4iY52mvYP2OI1c5dV5ydnS3WJ0+eTEJCAjNnzqRNmzY4Ojry0EMPUVZWds3j2NvbW6yrVCqMRmON6jf0ZI0hISGkpqaycuVKEhISePbZZ/nwww9Zu3Ytrq6u7Nq1izVr1vDXX38xdepUpk+fzvbt223uETC5oq4rEXdB+FDQOl+/rhDC5qlUKpy0dlZZ6vMNaRs3bmTs2LHcf//9dOrUiYCAAE6cOFFv56uOu7s7/v7+bN++3VxmMBjYtWtXjY7Trl07Nm60vOW4ceNG2rdvb153dHRk+PDhfPrpp6xZs4bNmzeTnJwMgJ2dHTExMXzwwQfs3buXEydOsGrVqpv4ZvVDrqjrytB3rR2BEEJcV1hYGL/++ivDhw9HpVLx5ptvXvPKuL4899xzxMXF0aZNG9q2bctnn33G+fPna/RHyssvv8wjjzxCly5diImJ4X//+x+//vqreRR7fHw8BoOBXr164eTkxH//+18cHR0JDQ3ljz/+4NixYwwYMABPT0+WLl2K0WgkIiKivr5yrUmiFkKIW8i//vUvnnjiCfr06YOPjw+vvvoqBQUFDR7Hq6++SlZWFo899hgajYbx48czZMiQGs0yNWLECD755BNmzpzJCy+8QMuWLZk/fz7R0dGAaT7o9957j0mTJmEwGOjUqRP/+9//8Pb2xsPDg19//ZXp06dTWlpKWFgYP/74Ix06dKinb1x7KqWhbxpY2cmTJwkJCSEjI4NmzZrd9PEqDEY0atXFvwLzMkBtB243N/+oEKLhlJaWcvz4cVq2bImDg4O1w7klGY1G2rVrxyOPPMLbb79t7XDqxLV+r2qSi+Qe9U14ZdEeur6dQMqpyr9Gl/8TZnWEbV9ZNzAhhLBxaWlpzJs3j0OHDpGcnMwzzzzD8ePH+dvf/mbt0GyOJOqbcP5COQWlFaw9VPmIgn8HUGngwlnrBiaEEDZOrVYTHx9Pjx496Nu3L8nJyaxcuZJ27dpZOzSbI/eob8LAcF8S9mez9lAuE+4Igw4joP29oHO1dmhCCGHTQkJCrhixLaonifomDAw3vbh+V3oe+SXluDvKo1lCCCHqlnR934QQLyda+zpjMCpsPHLGcqMVHncQQgjR9EiivkkDw/0AWJuaayo4tRPm3QHf3mvFqIQQQjQVkqhv0sAIU/f32kO5ptfjOXiYknXGVigvsW5wQgghGj1J1DepV0svdHZqsgpKSc0uBK9W4BoIhjI4uf36BxBCCCGuwaqJOi4ujh49euDq6oqfnx8jRowgNTX1uvstXLiQtm3b4uDgQKdOnWo8NVpdcrDX0Lu1N1DZ/a1Smaa9BDghIxqFEELcHKsm6rVr1xIbG8uWLVtISEigvLycwYMHU1xcfNV9Nm3axKhRoxg3bhy7d+9mxIgRjBgxgpSUlAaM3FLV6O+1hyrvU1dNe3lig5UiEkKIGxcdHc3EiRPN6y1atGDWrFnX3EelUrFkyZKbPnddHedapk+fTufOnev1HPXJqol6+fLljB07lg4dOhAVFUV8fDzp6ens3Lnzqvt88sknDB06lJdffpl27drx9ttv07VrV2bPnt2AkVuqStTbT5yjWF9x8Yr65HYoL7VaXEKIpm348OEMHTq02m3r169HpVKxd+/eGh93+/btjB8//mbDs3C1ZJmZmcmwYcPq9FxNjU3do87PzwfAy8vrqnU2b95MTEyMRdmQIUPYvHlztfX1ej0FBQXmpbCwsO4CrtTSx5nmXk6UGxQ2HT0L3m3AxR8MetPAMiGEqAfjxo0jISGBkydPXrFt/vz5dO/encjIyBof19fXFycnp7oI8boCAgLQ6XQNcq7GymYStdFoZOLEifTt25eOHTtetV5WVhb+/v4WZf7+/mRlZVVbPy4uDnd3d/Ny6TyldUWlUl3S/Z1juk8t3d9CiHp2zz334OvrS3x8vEV5UVERCxcuZNy4cZw9e5ZRo0YRHByMk5MTnTp14scff7zmcS/v+j58+DADBgzAwcGB9u3bk5CQcMU+r776KuHh4Tg5OdGqVSvefPNNysvLAdN0k2+99RZ79uxBpTJNYlQV8+Vd38nJydxxxx04Ojri7e3N+PHjKSoqMm8fO3YsI0aMYObMmQQGBuLt7U1sbKz5XDfCaDQyY8YMmjVrhk6no3Pnzixfvty8vaysjAkTJhAYGIiDgwOhoaHExcUBoCgK06dPp3nz5uh0OoKCgnj++edv+Ny1YTOJOjY2lpSUFBYsWFCnx50yZQr5+fnmZf/+/XV6/CpViXpNauVjWi0qE3WaJGohGrWy4povhoqL+xsqTGWXP655tX1rwM7Ojscee4z4+HgunQhx4cKFGAwGRo0aRWlpKd26dePPP/8kJSWF8ePH8+ijj7Jt27YbOofRaOSBBx5Aq9WydetWvvjiC1599dUr6rm6uhIfH8/+/fv55JNPmDdvHh9//DEAI0eO5KWXXqJDhw5kZmaSmZnJyJEjrzhGcXExQ4YMwdPTk+3bt7Nw4UJWrlzJhAkTLOqtXr2ao0ePsnr1av7zn/8QHx9/xR8r1/LJJ5/w0UcfMXPmTPbu3cuQIUO49957OXz4MACffvopv//+Oz///DOpqal8//33tGjRAoBffvmFjz/+mC+//JLDhw+zZMkSOnXqdMPnrg2beIXohAkT+OOPP1i3bt11p/sKCAggOzvboiw7O5uAgIBq6+t0Ootulfqad7V3a2+0GjUnz5dw7EwxrUMr71NnbIcKPdhJ144QjdK7QTXf5+F46HC/6fPB/8HCsRDaDx7/82KdWZ2qn8Bnen6NTvXEE0/w4YcfsnbtWvM8zPPnz+fBBx809yROnjzZXP+5555jxYoV/Pzzz/Ts2fO6x1+5ciUHDx5kxYoVBAWZ2uLdd9+94r7yG2+8Yf7cokULJk+ezIIFC3jllVdwdHTExcUFOzu7q/5bDfDDDz9QWlrKt99+i7Oz6ZXMs2fPZvjw4bz//vvm3lRPT09mz56NRqOhbdu23H333SQmJvLUU0/dUJvNnDmTV199lf/7v/8D4P3332f16tXMmjWLOXPmkJ6eTlhYGP369UOlUhEaGmreNz09nYCAAGJiYrC3t6d58+Y31I43w6pX1IqiMGHCBBYvXsyqVato2bLldffp3bs3iYmJFmUJCQn07t27vsK8Ic46O3q09AQqH9PyjQAnH6gogVO7rBqbEKLpatu2LX369OGbb74B4MiRI6xfv55x48YBYDAYePvtt+nUqRNeXl64uLiwYsUK0tPTb+j4Bw4cICQkxJykgWr/vf3pp5/o27cvAQEBuLi48MYbb9zwOS49V1RUlDlJA/Tt2xej0Wjx6G6HDh3QaDTm9cDAQHJycm7oHAUFBZw+fZq+fftalPft25cDBw4Apu71pKQkIiIieP755/nrr7/M9R5++GFKSkpo1aoVTz31FIsXL6aiooL6ZNUr6tjYWH744Qd+++03XF1dzfeZ3d3dcXR0BOCxxx4jODjYfH/ghRdeYODAgXz00UfcfffdLFiwgB07dvDVV9afA3pguC8bj5xl7aFcnujX0tT9vf83U/d3qHX/kBBC1NI/T9d8H80lPWhth5uOobrsumhi8s3FdYlx48bx3HPPMWfOHObPn0/r1q0ZOHAgAB9++CGffPIJs2bNolOnTjg7OzNx4kTKysrq7PybN29m9OjRvPXWWwwZMgR3d3cWLFjARx99VGfnuJS9vb3FukqlwliH8yt07dqV48ePs2zZMlauXMkjjzxCTEwMixYtIiQkhNTUVFauXElCQgLPPvusuUfj8rjqilWvqOfOnUt+fj7R0dEEBgaal59++slcJz09nczMTPN6nz59+OGHH/jqq6+Iiopi0aJFLFmy5JoD0BpKdITpvd9bjp2ltNxg6uoCU/e3EKJx0jrXfNFccg2ksTOV2Tve2HFr4ZFHHkGtVvPDDz/w7bff8sQTT6BSqQDYuHEj9913H3//+9+JioqiVatWHDp06IaP3a5dOzIyMiz+Hd6yZYtFnU2bNhEaGsrrr79O9+7dCQsLIy0tzfLrarUYDIbrnmvPnj0W79LYuHEjarWaiIiIG475Wtzc3AgKCrpiis2NGzdaDDZ2c3Nj5MiRzJs3j59++olffvmFc+fOAeDo6Mjw4cP59NNPWbNmDZs3byY5ue7+8LqcVa+oLx38cDVr1qy5ouzhhx/m4YcfroeIbk6YnwuB7g5k5pey5dhZotvfB8HdIDDK2qEJIZowFxcXRo4cyZQpUygoKGDs2LHmbWFhYSxatIhNmzbh6enJv/71L7Kzs2/4CZiYmBjCw8MZM2YMH374IQUFBbz++usWdcLCwkhPT2fBggX06NGDP//8k8WLF1vUadGiBcePHycpKYlmzZrh6up6xWNZo0ePZtq0aYwZM4bp06eTm5vLc889x6OPPnrF0z434+WXX2batGm0bt2azp07M3/+fJKSkvj+++8B+Ne//kVgYCBdunRBrVazcOFCAgIC8PDwID4+HoPBQK9evXBycuK///0vjo6OFvex65rNjPpuCiwf08oFV39o1s3yr2shhKgH48aN4/z58wwZMsTifvIbb7xB165dGTJkCNHR0QQEBDBixIgbPq5arWbx4sWUlJTQs2dPnnzySd555x2LOvfeey8vvvgiEyZMoHPnzmzatIk333zTos6DDz7I0KFDuf322/H19a32ETEnJydWrFjBuXPn6NGjBw899BCDBg2q8xdaPf/880yaNImXXnqJTp06sXz5cn7//XfCwsIA0wj2Dz74gO7du9OjRw9OnDjB0qVLUavVeHh4MG/ePPr27UtkZCQrV67kf//7H97e3nUa46VUyo1c1jYhJ0+eJCQkhIyMjOuOMK+NZcmZPPP9Llr5OrPqpeg6P74Qou6VlpZy/PhxWrZsiYODg7XDEU3EtX6vapKL5FKvjvUN80GjVnEst5iMcxcIMZyEzZ+BSgPDZ1k7PCGEEI2MdH3XMTcHe7o1Nz2mteZQruk1oru+heSFli9BEEIIIW6AJOp6MDCi8j51ai74dYB+k+Chb4Bb6i6DEEKIOiCJuh5UDSjbdPQMeqMCMdMgfAho6ucZOyGEEE2XJOp60D7QDR8XHRfKDOw8cd7a4QghhGjEJFHXA7VaxYBwH6DyMS2jAY4kwqp3TJ+FEDapLt9uJURd/T7JqO96Eh3hx6+7TrH2UC5ThobDwsdBnw9t74KgLtYOTwhxCa1Wi1qt5vTp0/j6+qLVas1v9hKiphRFoaysjNzcXNRqNVqt9qaOJ4m6nvRv44NKBQezCsksLCMwtDccWg4nNkqiFsLGqNVqWrZsSWZmJqdP1+Ld3kJUw8nJiebNm6NW31zntSTqeuLprCWqmQdJGXmsO5TLyNC+lYl6A/SZcP0DCCEalFarpXnz5lRUVFz3ndRCXI9Go8HOzq5OemYkUdejgeG+JGXksfZQLiOjK6dUS99kuk+t1lx7ZyFEg1OpVNjb29fbLEhC1IYMJqtH0ZXPU68/fIYKv06gdYXSfMjeZ+XIhBBCNBaSqOtRZDMPPJzsKSytYPepImh+m2nDiQ3WDUwIIUSjIYm6HmnUKvqHXfKWshaV3d9pG6+xlxBCCHGRJOp6Fl35lrI1h3IgtJ+pMG0jyPOaQgghboAk6nrWv/LFJymnCsh1bQf2zlByHnL2WzkyIYQQjYEk6nrm5+pAhyA3ANYfy4PmvUwbpPtbCCHEDZBE3QCqRn+vPZQLoZX3qWVAmRBCiBsgiboBDAz3A2DdoVwMl96nVmTaSyGEENcmLzxpAF2ae+Cqs+P8hXJSlFZEhQ0xdYFX6MHewdrhCSGEsGGSqBuAvUZNvzAflqVkseZIPlGjf7Z2SEIIIRoJ6fpuIAMvfUxLCCGEuEGSqBvIgMpEvScjj/PFZVCYDfuWyH1qIYQQ1ySJuoEEeTgS7u+CUYGNh07DJ5GwcAycPWLt0IQQQtgwqybqdevWMXz4cIKCglCpVCxZsuSa9desWYNKpbpiycrKapiAb1J0hGn095oj+RDSCwIi4cI5K0clhBDCllk1URcXFxMVFcWcOXNqtF9qaiqZmZnmxc/Pr54irFtV96nXHsrFOPoXeHr9xRegCCGEENWw6qjvYcOGMWzYsBrv5+fnh4eHR90HVM+6t/DESasht1DPgZwLdAhyt3ZIQgghbFyjvEfduXNnAgMDufPOO9m4sfG8ilNnp6FPa2+g8i1lAOUlUHbBilEJIYSwZY0qUQcGBvLFF1/wyy+/8MsvvxASEkJ0dDS7du266j56vZ6CggLzUlhY2IARX8n8mFZqLix9Bd5rDskLrRqTEEII29WoXngSERFBRESEeb1Pnz4cPXqUjz/+mO+++67afeLi4njrrbcaKsTrMr1OdB+70s6jb+mCzlBmep1otzHWDk0IIYQNalRX1NXp2bMnR45c/RGnKVOmkJ+fb17277fu9JLNvZ1o5eNMhVFhj6aTqfDEBnmeWgghRLUafaJOSkoiMDDwqtt1Oh1ubm7mxdXVtQGjq17Vy0/+ON8M1PZQcArOn7BuUEIIIWySVRN1UVERSUlJJCUlAXD8+HGSkpJIT08HTFfDjz32mLn+rFmz+O233zhy5AgpKSlMnDiRVatWERsba43wa21g5bSXKw8XoAR3NRXKtJdCCCGqYdV71Dt27OD22283r0+aNAmAMWPGEB8fT2ZmpjlpA5SVlfHSSy9x6tQpnJyciIyMZOXKlRbHaAx6t/JGZ6fmdH4p5zv0xCtjq+k+dddHrR2aEEIIG6NSlFvr5ujJkycJCQkhIyODZs2aWS2Ox77ZxrpDucy9LY9hSc+Ce3N4Mdlq8QghhGg4NclFjf4edWNV9ZjWopxgUGkgPx3Op1k5KiGEELZGErWVVCXq9WklGIK6mArTGs/LW4QQQjSMWiXqjIwMTp48aV7ftm0bEydO5KuvvqqzwJq61r7ONPN0pMxg5KRbZaI+IYlaCCGEpVol6r/97W+sXr0agKysLO688062bdvG66+/zowZM+o0wKZKpVKZr6rX6Stf4nJivRUjEkIIYYtqlahTUlLo2bMnAD///DMdO3Zk06ZNfP/998THx9dlfE1aVaL+ISvIdJ86Lw3yT15nLyGEELeSWiXq8vJydDodACtXruTee+8FoG3btmRmZtZddE1cnzY+2GtUHDgHet9OYOcAuanWDksIIYQNqVWi7tChA1988QXr168nISGBoUOHAnD69Gm8vb3rNMCmzEVnR/dQLwD+FxEHr6VDm0FWjkoIIYQtqVWifv/99/nyyy+Jjo5m1KhRREVFAfD777+bu8TFjal6S9mf6XZgp7NyNEIIIWxNrd5MFh0dzZkzZygoKMDT09NcPn78eJycnOosuFtBdIQv7y07yOZjZyktN+BgrzFN0KFSWTs0IYQQNqBWV9QlJSXo9Xpzkk5LS2PWrFmkpqbi5+dXpwE2dRH+rvi76SgtN3J66Qcw5zZI+cXaYQkhhLARtUrU9913H99++y0AeXl59OrVi48++ogRI0Ywd+7cOg2wqbv0Ma3s02mQe0Am6BBCCGFWq0S9a9cu+vfvD8CiRYvw9/cnLS2Nb7/9lk8//bROA7wVREeYeiG+KboNHvkO7njTyhEJIYSwFbVK1BcuXDDP6/zXX3/xwAMPoFarue2220hLk/dV11TfNj5o1CoSzvpyMjAGnGXkvBBCCJNaJeo2bdqwZMkSMjIyWLFiBYMHDwYgJycHNze3Og3wVuDuaE+XEA8A1h06Y91ghBBC2JRaJeqpU6cyefJkWrRoQc+ePenduzdgurru0qVLnQZ4q6i6T70/eSeseQ+2fmnliIQQQtiCWiXqhx56iPT0dHbs2MGKFSvM5YMGDeLjjz+us+BuJVX3qYsykmFNHOz4xsoRCSGEsAW1eo4aICAggICAAPMsWs2aNZOXndyEDkFueDtrWVscBg5A7kEoPgPOPtYOTQghhBXV6oraaDQyY8YM3N3dCQ0NJTQ0FA8PD95++22MRmNdx3hLUKtVDAj35Txu5Di2NhXK/NRCCHHLq1Wifv3115k9ezbvvfceu3fvZvfu3bz77rt89tlnvPmmPFpUW9GVrxPdYmxnKpDnqYUQ4pZXq67v//znP3z99dfmWbMAIiMjCQ4O5tlnn+Wdd96pswBvJf3a+KBSwbLC1tyrBU7IFbUQQtzqanVFfe7cOdq2bXtFedu2bTl37txNB3Wr8nbRERnszjZjZdvm7IML0p5CCHErq1WijoqKYvbs2VeUz549m8jIyJsO6lY2MMKPs7iTqQ01FaRtsm5AQgghrKpWXd8ffPABd999NytXrjQ/Q71582YyMjJYunRpnQZ4qxkY7suniYdZVxbBSNJM96nb3WPtsIQQQlhJra6oBw4cyKFDh7j//vvJy8sjLy+PBx54gH379vHdd9/VdYy3lKhm7rg72rO+LMJUkCYDyoQQ4lZW6+eog4KCrhg0tmfPHv7973/z1Vdf3XRgtyo7jZp+YT5s3Vs58jsrBUrOg6PntXcUQgjRJNXqilrUr+hwX3Lx4KSmGaBA2mZrhySEEMJKrJqo161bx/DhwwkKCkKlUrFkyZLr7rNmzRq6du2KTqejTZs2xMfH13ucDa3qvd/rysJNBfLiEyGEuGVZNVEXFxcTFRXFnDlzbqj+8ePHufvuu7n99ttJSkpi4sSJPPnkkxbvG28K/NwcaBfoxv8MvTkQEQsdH7R2SEIIIaykRveoH3jggWtuz8vLq9HJhw0bxrBhw264/hdffEHLli356KOPAGjXrh0bNmzg448/ZsiQITU6t62LjvBlbmYHvlIH83FwZ2uHI4QQwkpqdEXt7u5+zSU0NJTHHnusvmJl8+bNxMTEWJQNGTKEzZub3j1cc/f3oVyMRsXK0QghhLCWGl1Rz58/v77iuCFZWVn4+/tblPn7+1NQUEBJSQmOjo5X7KPX69Hr9eb1wsLCeo+zLnQL9cRFZ0d58TkyNv1MqI8rtL3L2mEJIYRoYE1+1HdcXJzFVX/79u2tHdINsdeo6dvGmzvUSYSuHA/rZ1o7JCGEEFbQqBJ1QEAA2dnZFmXZ2dm4ublVezUNMGXKFPLz883L/v37GyLUOjEw3I+txnZkaEIguDso0gUuhBC3mkaVqHv37k1iYqJFWUJCgvk1ptXR6XS4ubmZF1dX1/oOs84MjPAlE28GXnif/Oh3QKWydkhCCCEamFUTdVFREUlJSSQlJQGmx6+SkpJIT08HTFfDlw5Oe/rppzl27BivvPIKBw8e5PPPP+fnn3/mxRdftEb49S7Yw5EwPxeMCmw4csba4QghhLACqybqHTt20KVLF7p06QLApEmT6NKlC1OnTgUgMzPTnLQBWrZsyZ9//klCQgJRUVF89NFHfP31103u0axLVY3+3nDwFGQlWzkaIYQQDU2lKLfWjc+TJ08SEhJCRkYGzZo1s3Y417X+cC4T/53ARocX0KmNqF5LB62ztcMSQghxE2qSixrVPepbUY8WXlyw9+KM4obKWAEZW60dkhBCiAYkidrGOdhr6N3am63GtqaCEzLtpRBC3EokUTcCA8N92WKsfP77hEzQIYQQtxJJ1I3AwHBfthpN81Mrp3ZC2QUrRySEEKKhSKJuBFr4OKP2bEGm4oXKWA4nt1s7JCGEEA1EEnUjMTDCjy2VV9Vyn1oIIW4dkqgbiYERl3R/p0miFkKIW4Uk6kbitlbe7FKZBpQpJ3dCeamVIxJCCNEQJFE3Ek5aO/xbdCBb8UBt0MOpHdYOSQghRAOQRN2IDIzwM3d/y31qIYS4NUiibkSiL7lPbTguiVoIIW4Fkqgbkda+Lhxz7kKZoiFfb5T5qYUQ4hYgiboRUalUtIjoTKT+az4N+lDmpxZCiFuAJOpGZmCEH6XoWHco19qhCCGEaACSqBuZvm28sVOrOHammIysM9YORwghRD2TRN3IuDrYE91MxR/afxIwrxNUlFk7JCGEEPVIEnUj1LVdGwJVZ7E3XIDsFGuHI4QQoh5Jom6EoiP8+UfZiww0zkXvH2XtcIQQQtQjSdSNULtAV9Jcokgrc2fHifPWDkcIIUQ9srN2AKLmVCoVA8N9WbTzJPo1H8GGZPDvCAEdIaAT+LYFO521wxRCCFEHJFE3UtERpkTtmLkVDDvhxPqLG9V24BN+MXn7VyZwFz/rBSyEEKJWJFE3Uv3a+GCnVjHtwiNEqrvTXp1ON90pwpQTOBkKIGe/aUn++eJOzn6mxB35fxA10nrBCyGEuGGSqBspDyctn47qwpLdfqzNaMOiQj2UAygEcI726jS6aE/S0+k0YcYTeJZmoCrOgaOroHmfiwc6nwY//R2Cu8HwWVb6NkIIIa5GEnUjdlenQO7qFIiiKJzOLyUpPY/d6edJyvBi4ylfVpV2hcppqx0pJUJ1kn6umRhPtMLf/gRdmnvQLn8v9ll7gcveG/5D5RW3ufu8E3i1BLWmQb+jEELc6iRRNwEqlYpgD0eCPRy5OzIQgHKDkYOZhSRlnGd3Rh5J6XkknXEgqaANFAAH9gHgb3eBB7zfoKWzM457TtOluQfBrnaojq4CQxkcWn7xRPZO4Nfe8r63b1tw9Kj376goCoX6Cs4WlXG2SM+ZojJKyivo0cKLZp5O9X5+IYSwFpWi3FpTMJ08eZKQkBAyMjJo1qyZtcNpUHkXykjKyDMvu9PzyC8pv6Kev7MdD/if5janTNpyHJ/iw2hyD0JFSfUHdvE3DV6LeQuadTOVGcpNg9quMXGIvsLAueIyzhaVcaZIb0rCxfrKddNnc3lRGWUGY7XHiWrmztCOgQzrGEALH+cat4sQQjS0muQim0jUc+bM4cMPPyQrK4uoqCg+++wzevbsWW3d+Ph4Hn/8cYsynU5HaWnpDZ3rVk7Ul1MUhRNnL1R2l5uS9/7TBVQYLX8lVCpo6+tEjH8Rtzln0laVhlfhIVTZKVB42lzP+ORq8j07crZYj2b7PEJ2f0hq8IOsaPY8Z4v0nC3Uo80/yv5Sb7KLDRSWVtQ4ZhedHd4uWrydtShAUkaexWyf7QLduKtjAMM6BdDGz7W2TSOEEPWqJrnI6l3fP/30E5MmTeKLL76gV69ezJo1iyFDhpCamoqfX/WPE7m5uZGammpeV8l0j7WiUqlo6eNMSx9nHuhq+kUpLTew73Q+u9PzzF3mp/JKOJBzgQM5aj4jGAjGWdufTs3ccXUtwbHgGJ4lJ/jl8xMUGTMBmGG3icfsLrDuaB6fph4GwJfzbHeIpVzRkKb4c9Q+iGMEk61tznmnllxwa4WLmyfezlq8XXR4u2jxcdHi7azDx1WHt7MWB3vLe+S5hXr+2p/FsuQsNh87y4HMAg5kFvBRwiHC/FwY1jGAYZ0CaRvgKr8nQohGyepX1L169aJHjx7Mnj0bAKPRSEhICM899xyvvfbaFfXj4+OZOHEieXl5tTqfXFHXXE5h5UC1ysS992QexWWGq9Z3d7TH31lFB4dzODq7ovFsjreLlnDDYQZvexI7w4Wrn8w1CHzDTV3pVUtIL7B3uG6c54vLSNifzdKUTDYeOUO54eKvdgtvJ4Z1MnWPdwp2l6QthLCqRtP1XVZWhpOTE4sWLWLEiBHm8jFjxpCXl8dvv/12xT7x8fE8+eSTBAcHYzQa6dq1K++++y4dOnSo9hx6vR69Xm9eP3XqFO3bt5dEfRMMRoXDOYUkn8zHTqPC27nq6leHp5MWrd013kxrNJq6y3NT4cxhOFP5MzcVinOq32fy4Ysva0n5FfLSIexO8K/+vzlAfkk5iQeyWZaSxdpDuZRVXLy/HezhaL7S7hLigVotSVsI0bAaTdf3mTNnMBgM+Pv7W5T7+/tz8ODBaveJiIjgm2++ITIykvz8fGbOnEmfPn3Yt29ftV82Li6Ot956q17iv1Vp1CraBrjRNsCt5jur1eDezLS0GWS5reR8ZfI+VJnID0FhFjj7Xqyz9yfTSHSt88VEfe447P6v6Vnw4G7g6o+7oz0PdG3GA12bUaSvYPXBHJalZLL6YC6n8kr4esNxvt5wnAA3B4Z2DGBoxwB6tPBCI0lbCGFjrHpFffr0aYKDg9m0aRO9e/c2l7/yyiusXbuWrVu3XvcY5eXltGvXjlGjRvH2229fsV2uqJuYbfMgfQv0ftaUlAF2fQe/T7hYxz0EgrteTNyBnUHnAkBJmYG1h3JYlpJF4oEcivQXB7T5uGgZ3CGAuzoG0quVF/YambNGCFE/Gs0VtY+PDxqNhuzsbIvy7OxsAgICbugY9vb2dOnShSNHjlS7XafTodNdnKCioKCg9gEL6+v5lGm5lHdr6PJ3OLULcg5AfoZp2V9560SlBt92ENwVx+BuDA3uxtCHO1FqVLHxyBmWJmeRsD+LM0Vl/LA1nR+2puPhZM/g9v4M6xhI3zY+1+7OF0KIemTVRK3VaunWrRuJiYnme9RGo5HExEQmTJhw7Z0rGQwGkpOTueuuu+oxUmHTQvuYFgB9IWTugZM74NROU/IuOAk5+0zL7u9M9YK64DB+DYPa+TOonT9leX5sztawfF8WK/Zlc664jJ93nOTnHSdxdbAjpp0/wzoGMCDc94qR50IIUZ+s/njWpEmTGDNmDN27d6dnz57MmjWL4uJi87PSjz32GMHBwcTFxQEwY8YMbrvtNtq0aUNeXh4ffvghaWlpPPnkk9b8GsJW6FyhRT/TUqUwqzJp77yYvC8diGYoRzu7MwO1Lgx8egNv39eRbcfPsSL5JEv3nyG3UM/i3adYvPsUTloNd7T1Y1jHQPq18cFJp8FOrZJR5EKIemP1RD1y5Ehyc3OZOnUqWVlZdO7cmeXLl5sHmKWnp6NWX+x2PH/+PE899RRZWVl4enrSrVs3Nm3aRPv27a31FYStcw2AtnebFjCNPC8vvrj93HEwGsBYDi7+2KnV9GnjQ5+kV5jumsS5kI5sK2/JL1n+rC8M5I+9mfyxN9PiFFqNGnuNCns7NfYa9cV1jWnd3k6N9tJ1jRqtnQo79cXPFtuq6tpdtn7JsXxddYT7u+LqYN+AjSmEaGhWf466oclz1KJa5aWmx758wy+WzYqEvDSLaka1PdmObdisD2VbSTOyFE9yFE+yFU/O4YpCw9/LDvZwJCLA1bT4m3628nVGZydd9ELYqkbzHLU1SKIWN+zCOTi929RVfmqH6b73hTNXra6o7cjt9zZn2v6dcoMRVV46Hkd+pcilBaeDh1FuMFJmMFJeYaTcqFBuMFJuqPxZYazcXlVeuV5x2bpBobzCdJxT50vIKqj+1bl2atNb58IDXGnr72r6GeBKiKeTPDcuhA1oNKO+hbBpTl6mZ72rnvdWFNNo8lM7TUk7NxWKsqAwG4pzURkr8PP1wy+o8vny4k2w52MI6kr7O8dePO7sHlB2wdQlf+niFQguVeuBpvNf59533oUyDmUXkZpVQGp2IalZhRzMKqSwtILDOUUcziniTy520zvaawj3dyEiwJVwf1faBrgRHuCCr4tO7rMLYaMkUQtxo1Qq8GhuWjrcb7nNUA5FOeBwyUtgXP2hy6Om+lUUBfIyTDORFZy89vnU9heTeP+XIGKYqbz4DJxOAo8QPHwj6NnSi54tvS45hUJWQSmpWYUXl+xCDucUUVJuYM/JfPaczLc4lZezlnB/F1Piruw+jwhwxUUn/0QIYW3yf6EQdUFjD+7BlmVVL1y53HM7TCPRC7OgMBOKsk0/CyuvzgszTV3sxvKLz4SXX/J+9PQt8NNoaNYTnky4WD7vDqjQo3LyItDRi0AnL6KdvKG5F7T1wuDgSWa5C0cKtezLsyc518ihnCJOnC3mXHEZW46dY8uxc5ZfwcORtgEXu87D/V1p7etS6+fKFUXBYFSoMF7+02j6aai+3HBZfQd7jbz+VdwyJFEL0ZBUqouvUL2WijLTu8+rEnpw14vb1Brw6wDebSz3yd5/9TnDAQ3QrHKJBtN84ffMorTT3ziSU8Tpw7vx3fdvDpQH8umFIWQVlHIqrwT3/AMcT9WyQHEhHxfUag2h3k442GuqTaLmpGtUMBgsy411OCKmta8z/xjYmhGdg+WFNKJJk8FkQjQFimIa+FZyDi6chwtnKz+fu+zzOdPnqiv0h76Bjg+aPu//HX5+1DRb2bi/yLtQRmpWIR1/7o1zqentgUZUFChOnFdcKMEBPfaUKlrTT0w/9Yo9i4392Gw0PaseyFnu0WwmR/HgN+PF59t7qA5ipzKgV+zRo6VCXbU4UKHSYlDrMKrt0WjUaNQq7NSqyp9qTueVUFj5+tdAdwee7N+K/+sRgrN01YtGQgaTCXGrUaksr7qvp7zElLQd3C+W+YTD7W+YZyrzcNLSq5U3uHqBUgr6fNQoeKiK8VAVX+XAJgMGDKOo40Ds1CqcTq7Hb8kPVPi0Y+rY6dip1Wg0Kpy+mob67OGrH8QIGFWAA6h0oHaEvi/Abc9QWFrOgs1HObjhV1bmt+btP0r5bNVhxvRuwdg+LfB01t54Wwhh4yRRC3Ersne88p66X1vTcrnYLaafhnLTDGfmq/ISqCitXPSV63qoKCWgTV/wM02EQkUziByJnWsg3i4X37uPVytTN/4l+5kXM8XUnV9RAqV55m2uDvY8FVYEa99H7+rBEPv5nDhXwieJh/lu3X7u6xnGU/1bEeThWGdNJoS1SKIWQtwYjb3partqbvAbFdARHvjqyvLRP1dfX1HAUHZZAtebkrXLJVPiluaDTwQ6nzASH7md5SlZfL76MF+ee5zS7VrWbmuHsXkf+gwaTstWETWLWQgbIveohRCNm6Hc9EcEoOSfQvXxla8TzrULRN2yL97t74AWfcEj9LrPqAtRn+QetRDi1qG5+K5zlXswvHIc0reQk5zIhSPrCSk9hG9FJhxeZFoAxS0YVWhf06xrLfqZRtBL4hY2ShK1EKJpcfKCtnfh19Y09e3Rk6dZ/df/qDi+gR6qA0SqjmFfcAqSfzYtAC/uu/jIXMl50LmDWh75ErZBErUQoklr3SyI1k/8g9N5j/H1+uM8ue0w7QwH6aU+yEBtKi0dS3BwDsQ8zO3Xf8DJbXDvbGh3jzVDtwlF+gqO5hRxJKeII7lFZJy7gJ1ahYO95pJFjeMlny/d5nhJmaO9Bt0ln+018sfQjZBELYS4JQR5ODJ1eHueu6MN/9ncjvmbTvDxhXJUF4z4vr+acf1a8reeIbhmJZuuqi8dFZ/yK+z50dRVHtoXAjuDXdN5BExRFM4UlZmTcVViPppbRGZ+9RO/1AWNWoWDnRpHrQadXWXC12pwsLP8I+DShO/lrKNvG286BrnfMm+mk8FkQohbUrG+ggXbM/h6/TFzMnJ1sGNsr2DGtc7Ho3Uv0FRey/wWC7v/e3FntZ3p8TKf8MuWNpbPptsYo1Hh5PkSjuQWmhJxTjFHck1JOb+k/Kr7+bjoaOPnTBs/F1p4OwNQUmagtMJAabmRknIDpeUG9Jd8Li03UFJuRG/+bKpbWmGgLrKOt7OWAeG+REf40j/MF69G9uy8THN5DZKohRCXKqsw8lvSKb5Ye5SjuaYXuejs1DzSPYTxA1oR4uUEOQfg6GpI22haSs5f/YAuAeATBm3vhtueuViuKA02YE1fYeD4mWJTIq68Sj6SU8Sx3CL0FcZq91GpoJmnI218XWjjd8ni64q7k321+9SGoijoK4zoK5O2RcKv/Ky/NLFf8llfbvpem46epajyzXRVsUc18yA6wpeB4b5ENvNAY+NX25Kor0EStRCiOkajQsKBbD5fc5Q9GXmAqWt2eGQgT0e3pm2AW1VFKDxtmub0zGE4c6hyOWya9rRK9yfgno9Nn8uKYWa46Sr8iRWgdTKVF2abrsDtHWoVc0FpucX946rP6ecuXPW96lqNmpY+pqvj1uZk7EIrX2cc7DW1iqOhlVUY2Zl2njWHclibmsvBrEKL7Z5O9uar7QFhvpYv2rERkqivQRK1EOJaFEVh87GzzF1zlPWHz5jL72jrxzPRrenRwuvqO5fmw5kjpsTt1RKa32Yqz9wDXw4AJx945SjlBlMXsW7BSLQnVlHuFkKJW2uKXVtR4NKS804tOKMLJV/lRmmF6UqzpPLKsqTMQPq5CxzJKSKnUH/VUFx1dhcTsZ8LrSuvlEM8HbFrYoO4svJLWXsohzWpuWw4fMb8HngwXW13CnYnOtyXgRF+dA6xjattSdTXIIlaCHGjUk7lM3ftUZYmZ5rvq3YP9eSeyEAqjKYu3EuTaOllCbWq27a8rByv8tO4lJ9jY3k4FZWXu39qp9BBnXbV859XXDiqBHHUGMQRJYijShDJxpbk4gmAjjLCXUoI8XbDO7CFOSlH2GfjrTOiUoygGE29AIoRFAMYDRc/X7rNu7VpASjJg6OJoNFZjnxPXWaahtVYeRxjxSXLVdZD+0CHEab9L5yDpZMBFTz074vHXfUOpG++xjHLL65rtKbn3sPuhF7/uKLNyg1GdqWdZ+2hXNak5rI/s8Biu7ujPf3DfIiO8GNguC++rta52pZEfQ2SqIUQNXX8TDFfrTvKLztPUWao/h5vbahUCs3si2hrl0WYOpPW6tO0UE4RYjyJjyEHNVf+87wxNJZTnZ4xJeSiHTj//BD4d4RnNl6s9GlXOHe0ZsHc/gYMfNn0OSsZvuhnemXr5EMX6/x7MGRsrdlxe/4D7vrA9LkwCz6KAJUGpl0y9/mC0XDwj5odt/NoGPG56XNFGXwSZfpD4/9+AIfK2xRlxeSUqFlz+AxrD+Wy/lAuBaUVFofpGOxGdLgfAyN86RLi0WC9DfJmMiGEqEMtfZyJeyCSiTHh/GfTCQ7nFOFY+ciQo/bi88KOWrXF88NXbr9YrrNXo7NTo7raALOyC6ZkW3X/u/JeeN8+AyAixFTnuAPYOVi8nQ0AZx8oKwKV2pQUVWrTC1ws1jWVn1Wmz5e+w13rAi36g6OH5XFD+5i679Ua08h3jb3pZ9W6eblkvVmPi/vr3GDoe6byS/WOhY4PXOUY9pZlZUWVtxZaXdz/3DHTuAF9IehcL5Yv/gd+x9byiE84j/hGYBgUxjGCWXvWi9/T7dh7upiUUwWknCpg9uojuDnY0T/Ml4ERvkSH++LnVruxA3VNrqiFEEI0bhV6yE6BolyIGHqxfM5tkHug+n00Oiq8WpNpH0qy3p815zzZU+rPcSWQMkx/+LQLdCO6Mml3DfWs0xe0SNf3NUiiFkKIW0SFHs4ehTOpkHuo8mflaH1D9QPxtjR7grjSB9l7Kh93pZA71LtJVUJI14bRL8yHgeG+3BMVhIvu5jqkpetbCCGEsNOBf3vTcimjAfLSLknehyD3IJw5xG29+vJbp36cLdJzcMOv9N3yBccI5o7SD1mWksWKfVkM6RAADTgGTRK1EEKIW4taY7rH7dXKsqtcUUwj4AFvFx19w4Mgqz8tvFqzpEtf1qTmkF1QimcDvwXNJh6mmzNnDi1atMDBwYFevXqxbdu2a9ZfuHAhbdu2xcHBgU6dOrF06dIGilQIIUSTVTWwrkqrgTD2D9T3fkLnEA8mxoQT90Bkg4dl9UT9008/MWnSJKZNm8auXbuIiopiyJAh5OTkVFt/06ZNjBo1inHjxrF7925GjBjBiBEjSElJaeDIhRBCiPpn9cFkvXr1okePHsyePRsAo9FISEgIzz33HK+99toV9UeOHElxcTF//HHxmbvbbruNzp0788UXX1z3fDKYTAghhLXVJBdZ9Yq6rKyMnTt3EhMTYy5Tq9XExMSwefPmavfZvHmzRX2AIUOGXLW+EEII0ZhZdTDZmTNnMBgM+Pv7W5T7+/tz8ODBavfJysqqtn5WVla19fV6PXr9xWH4hYWF1dYTQgghbJHV71HXt7i4ONzd3c1L+/btr7+TEEIIYSOsmqh9fHzQaDRkZ2dblGdnZxMQEFDtPgEBATWqP2XKFPLz883L/v376yZ4IYQQogFYtetbq9XSrVs3EhMTGTFiBGAaTJaYmMiECROq3ad3794kJiYyceJEc1lCQgK9e/eutr5Op0Onu/hkel5eHgCZmZl18h2EEEKImqrKQUbjDUzyoljZggULFJ1Op8THxyv79+9Xxo8fr3h4eChZWVmKoijKo48+qrz22mvm+hs3blTs7OyUmTNnKgcOHFCmTZum2NvbK8nJyTd0vm3btimALLLIIossslh92bZt23XzltXfTDZy5Ehyc3OZOnUqWVlZdO7cmeXLl5sHjKWnp6NWX+yh79OnDz/88ANvvPEG//znPwkLC2PJkiV07Njxhs7XpUsXtm3bhr+/v8Vxa6OwsJD27duzf/9+XF1dr7/DLU7aq+akzWpG2qtmpL1qpi7by2g0kp2dTZcuXa5b1+rPUTdmBQUFuLu7k5+fj5ubm7XDsXnSXjUnbVYz0l41I+1VM9ZqryY/6lsIIYRozCRRCyGEEDZMEvVN0Ol0TJs2zWJUubg6aa+akzarGWmvmpH2qhlrtZfcoxZCCCFsmFxRCyGEEDZMErUQQghhwyRRCyGEEDZMEvVNmDNnDi1atMDBwYFevXqxbds2a4dks9atW8fw4cMJCgpCpVKxZMkSa4dks+Li4ujRoweurq74+fkxYsQIUlNTrR2WzZo7dy6RkZG4ubnh5uZG7969WbZsmbXDajTee+89VCqVxWuZhaXp06ejUqkslrZt2zbY+SVR19JPP/3EpEmTmDZtGrt27SIqKoohQ4aQk5Nj7dBsUnFxMVFRUcyZM8faodi8tWvXEhsby5YtW0hISKC8vJzBgwdTXFxs7dBsUrNmzXjvvffYuXMnO3bs4I477uC+++5j37591g7N5m3fvp0vv/ySyMhIa4di8zp06EBmZqZ52bBhQ8OdvOZv5xaKoig9e/ZUYmNjzesGg0EJCgpS4uLirBhV4wAoixcvtnYYjUZOTo4CKGvXrrV2KI2Gp6en8vXXX1s7DJtWWFiohIWFKQkJCcrAgQOVF154wdoh2axp06YpUVFRVju/XFHXQllZGTt37iQmJsZcplariYmJYfPmzVaMTDRF+fn5AHh5eVk5EttnMBhYsGABxcXFV51RT5jExsZy9913W/w7Jq7u8OHDBAUF0apVK0aPHk16enqDndvqk3I0RmfOnMFgMJgnDqni7+/PwYMHrRSVaIqMRiMTJ06kb9++NzzxzK0oOTmZ3r17U1paiouLC4sXL6Z9+/bWDstmLViwgF27drF9+3Zrh9Io9OrVi/j4eCIiIsjMzOStt96if//+pKSkNMhkJpKohbBhsbGxpKSkNOz9sEYoIiKCpKQk8vPzWbRoEWPGjGHt2rWSrKuRkZHBCy+8QEJCAg4ODtYOp1EYNmyY+XNkZCS9evUiNDSUn3/+mXHjxtX7+SVR14KPjw8ajYbs7GyL8uzsbAICAqwUlWhqJkyYwB9//MG6deto1qyZtcOxaVqtljZt2gDQrVs3tm/fzieffMKXX35p5chsz86dO8nJyaFr167mMoPBwLp165g9ezZ6vR6NRmPFCG2fh4cH4eHhHDlypEHOJ/eoa0Gr1dKtWzcSExPNZUajkcTERLkvJm6aoihMmDCBxYsXs2rVKlq2bGntkBodo9GIXq+3dhg2adCgQSQnJ5OUlGReunfvzujRo0lKSpIkfQOKioo4evQogYGBDXI+uaKupUmTJjFmzBi6d+9Oz549mTVrFsXFxTz++OPWDs0mFRUVWfz1efz4cZKSkvDy8qJ58+ZWjMz2xMbG8sMPP/Dbb7/h6upKVlYWAO7u7jg6Olo5OtszZcoUhg0bRvPmzSksLOSHH35gzZo1rFixwtqh2SRXV9crxjs4Ozvj7e0t4yCuYvLkyQwfPpzQ0FBOnz7NtGnT0Gg0jBo1qkHOL4m6lkaOHElubi5Tp04lKyuLzp07s3z58isGmAmTHTt2cPvtt5vXJ02aBMCYMWOIj4+3UlS2ae7cuQBER0dblM+fP5+xY8c2fEA2Licnh8cee4zMzEzc3d2JjIxkxYoV3HnnndYOTTQRJ0+eZNSoUZw9exZfX1/69evHli1b8PX1bZDzy+xZQgghhA2Te9RCCCGEDZNELYQQQtgwSdRCCCGEDZNELYQQQtgwSdRCCCGEDZNELYQQQtgwSdRCCCGEDZNELYQQQtgwSdRCiHqjUqlYsmSJtcMQolGTRC1EEzV27FhUKtUVy9ChQ60dmhCiBuRd30I0YUOHDmX+/PkWZTqdzkrRCCFqQ66ohWjCdDodAQEBFounpydg6paeO3cuw4YNw9HRkVatWrFo0SKL/ZOTk7njjjtwdHTE29ub8ePHU1RUZFHnm2++oUOHDuh0OgIDA5kwYYLF9jNnznD//ffj5OREWFgYv//+u3nb+fPnGT16NL6+vjg6OhIWFnbFHxZC3OokUQtxC3vzzTd58MEH2bNnD6NHj+b//u//OHDgAADFxcUMGTIET09Ptm/fzsKFC1m5cqVFIp47dy6xsbGMHz+e5ORkfv/9d9q0aWNxjrfeeotHHnmEvXv3ctdddzF69GjOnTtnPv/+/ftZtmwZBw4cYO7cufj4+DRcAwjRGChCiCZpzJgxikajUZydnS2Wd955R1EURQGUp59+2mKfXr16Kc8884yiKIry1VdfKZ6enkpRUZF5+59//qmo1WolKytLURRFCQoKUl5//fWrxgAob7zxhnm9qKhIAZRly5YpiqIow4cPVx5//PG6+cJCNFFyj1qIJuz22283z29dxcvLy/y5d+/eFtt69+5NUlISAAcOHCAqKgpnZ2fz9r59+2I0GklNTUWlUnH69GkGDRp0zRgiIyPNn52dnXFzcyMnJweAZ555hgcffJBdu3YxePBgRowYQZ8+fWr1XYVoqiRRC9GEOTs7X9EVXVccHR1vqJ69vb3Fukqlwmg0AjBs2DDS0tJYunQpCQkJDBo0iNjYWGbOnFnn8QrRWMk9aiFuYVu2bLlivV27dgC0a9eOPXv2UFxcbN6+ceNG1Go1ERERuLq60qJFCxITE28qBl9fX8aMGcN///tfZs2axVdffXVTxxOiqZEraiGaML1eT1ZWlkWZnZ2decDWwoUL6d69O/369eP7779n27Zt/Pvf/wZg9OjRTJs2jTFjxjB9+nRyc3N57rnnePTRR/H39wdg+vTpPP300/j5+TFs2DAKCwvZuHEjzz333A3FN3XqVLp160aHDh3Q6/X88ccf5j8UhBAmkqiFaMKWL19OYGCgRVlERAQHDx4ETCOyFyxYwLPPPktgYCA//vgj7du3B8DJyYkVK1bwwgsv0KNHD5ycnHjwwQf517/+ZT7WmDFjKC0t5eOPP2by5Mn4+Pjw0EMP3XB8Wq2WKVOmcOLECRwdHenfvz8LFiyog28uRNOhUhRFsXYQQoiGp1KpWLx4MSNGjLB2KEKIa5B71EIIIYQNk0QthBBC2DC5Ry3ELUruegnROMgVtRBCCGHDJFELIYQQNkwStRBCCGHDJFELIYQQNkwStRBCCGHDJFELIYQQNkwStRBCCGHDJFELIYQQNkwStRBCCGHD/j/FTPccAOpuygAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 500x300 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "epochs_tensor = torch.linspace(0, num_epochs, len(train_losses))\n",
    "examples_seen_tensor = torch.linspace(0, examples_seen, len(train_losses))\n",
    "\n",
    "plot_values(epochs_tensor, examples_seen_tensor, train_losses, val_losses)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "564b4082",
   "metadata": {},
   "source": [
    "- 根据上述下降趋势可见模型学习效果良好；\n",
    "- 训练损失与验证损失非常接近，表明模型没有过拟合训练数据的倾向；\n",
    "- 同理，我们可以绘制下方的准确率图表；\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "id": "154a72db",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAeEAAAEiCAYAAADONmoUAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjMsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvZiW1igAAAAlwSFlzAAAPYQAAD2EBqD+naQAAXQdJREFUeJzt3XlYVNX/wPH3DDjsqyCCIqLirogbYW65hEskZmlmiUv601wz0yz3FsrKLDVNLW1zT81vuES47ysqLuSCogi4y6JsM/f3x+ToCCqD6CB8Xs8zzzNz7rnnfuaIfLj3nnuOSlEUBSGEEEI8dWpzByCEEEKUVJKEhRBCCDORJCyEEEKYiSRhIYQQwkwkCQshhBBmIklYCCGEMBNJwkIIIYSZSBIWQgghzESSsBBCCGEmkoSFEHlq2bIlw4cPN3cYQhRrkoSFeEJ69eqFSqXK9WrXrp25QxNCFBGW5g5AiOKsXbt2zJ8/36jMysrKTNEIIYoaORMW4gmysrKibNmyRi8XFxcANm3ahEajYevWrYb6U6ZMoUyZMiQnJwOwbt06mjZtirOzM6VLl+all17i9OnThvpnz55FpVKxdOlSmjVrho2NDY0aNeLff/9l7969NGzYEHt7e9q3b8/ly5cN+/Xq1YvQ0FAmTZqEu7s7jo6ODBgwgKysrAd+l8zMTEaOHEm5cuWws7MjMDCQTZs2GbafO3eOkJAQXFxcsLOzo1atWqxZs+aB7X3//ff4+flhbW2Nh4cHr776qmGbTqcjPDwcX19fbGxs8Pf3Z/ny5Ub7x8TE0L59e+zt7fHw8OCtt97iypUrhu0tW7Zk6NChjBo1CldXV8qWLcvEiRMfGI8Q5iBJWAgzuXPP9a233uLmzZscPHiQcePGMW/ePDw8PABIT09nxIgR7Nu3j6ioKNRqNZ07d0an0xm1NWHCBMaOHcuBAwewtLTkjTfeYNSoUXz77bds3bqVU6dOMX78eKN9oqKiOH78OJs2bWLRokWsWLGCSZMmPTDewYMHs3PnThYvXszhw4d57bXXaNeuHSdPngRg0KBBZGZmsmXLFo4cOcIXX3yBvb19nm3t27ePoUOHMnnyZGJjY1m3bh3Nmzc3bA8PD+eXX35h9uzZHD16lHfffZc333yTzZs3A3Djxg1atWpFQEAA+/btY926dSQnJ9O1a1ej4/z888/Y2dmxe/dupkyZwuTJk4mMjMznv5AQT4EihHgiwsLCFAsLC8XOzs7o9emnnxrqZGZmKvXq1VO6du2q1KxZU+nXr99D27x8+bICKEeOHFEURVHi4uIUQJk3b56hzqJFixRAiYqKMpSFh4cr1apVM4rN1dVVSU9PN5TNmjVLsbe3V7RaraIoitKiRQtl2LBhiqIoyrlz5xQLCwslISHBKJ7WrVsrY8aMURRFUerUqaNMnDgxX33zxx9/KI6OjkpKSkqubRkZGYqtra2yY8cOo/K+ffsq3bt3VxRFUT7++GPlxRdfNNp+/vx5BVBiY2MN8Tdt2tSoTqNGjZTRo0fnK0Yhnga5JyzEE/TCCy8wa9YsozJXV1fDe41Gw++//07dunXx8fHhm2++Map78uRJxo8fz+7du7ly5YrhDDg+Pp7atWsb6tWtW9fw/s5ZdJ06dYzKLl26ZNS2v78/tra2hs9BQUGkpaVx/vx5fHx8jOoeOXIErVZL1apVjcozMzMpXbo0AEOHDmXgwIH8/ffftGnThi5duhjFda+2bdvi4+NDpUqVaNeuHe3ataNz587Y2tpy6tQpbt26Rdu2bY32ycrKIiAgAIBDhw6xcePGPM+0T58+bYjz/uN7enrm6gchzEmSsBBPkJ2dHVWqVHlonR07dgBw7do1rl27hp2dnWFbSEgIPj4+zJ07Fy8vL3Q6HbVr185177ZUqVKG9yqVKs+y+y9hmyItLQ0LCwv279+PhYWF0bY7ifDtt98mODiYiIgI/v77b8LDw/n6668ZMmRIrvYcHBw4cOAAmzZt4u+//2b8+PFMnDiRvXv3kpaWBkBERATlypUz2u/OoLa0tDRCQkL44osvcrXt6elpeH9vH8Dj94MQhU2SsBBmdPr0ad59913mzp3LkiVLCAsL459//kGtVnP16lViY2OZO3cuzZo1A2Dbtm2FduxDhw5x+/ZtbGxsANi1axf29vZ4e3vnqhsQEIBWq+XSpUuGWPLi7e3NgAEDGDBgAGPGjGHu3Ll5JmEAS0tL2rRpQ5s2bZgwYQLOzs5s2LCBtm3bYmVlRXx8PC1atMhz3/r16/PHH39QsWJFLC3l15h4dslPrxBPUGZmJklJSUZllpaWuLm5odVqefPNNwkODqZ37960a9eOOnXq8PXXX/P+++/j4uJC6dKlmTNnDp6ensTHx/PBBx8UWmxZWVn07duXsWPHcvbsWSZMmMDgwYNRq3OP16xatSo9evSgZ8+efP311wQEBHD58mWioqKoW7cuHTt2ZPjw4bRv356qVaty/fp1Nm7cSI0aNfI89l9//cWZM2do3rw5Li4urFmzBp1OR7Vq1XBwcGDkyJG8++676HQ6mjZtys2bN9m+fTuOjo6EhYUxaNAg5s6dS/fu3Q2jn0+dOsXixYuZN29errN1IYoqScJCPEHr1q0zujwKUK1aNU6cOMGnn37KuXPn+OuvvwD9ZdQ5c+bQvXt3XnzxRfz9/Vm8eDFDhw6ldu3aVKtWje+++46WLVsWSmytW7fGz8+P5s2bk5mZSffu3R/6CM/8+fP55JNPeO+990hISMDNzY3nnnuOl156CQCtVsugQYO4cOECjo6OtGvXLtc97jucnZ1ZsWIFEydOJCMjAz8/PxYtWkStWrUA+Pjjj3F3dyc8PJwzZ87g7OxM/fr1+fDDDwHw8vJi+/btjB49mhdffJHMzEx8fHxo165dnn9ECFFUqRRFUcwdhBDi6erVqxc3btxg1apV5g5FiBJN/mQUQgghzESSsBBCCGEmcjlaCCGEMBM5ExZCCCHMRJKwEEIIYSaShIUQQggzkSRcQDNnzqRixYpYW1sTGBjInj17zB3SE7FlyxZCQkLw8vJCpVLleqRFURTGjx+Pp6cnNjY2tGnTxrCqzh3Xrl2jR48eODo64uzsTN++fQ1TE95x+PBhmjVrhrW1Nd7e3kyZMuVJf7XHFh4eTqNGjXBwcKBMmTKEhoYSGxtrVCcjI4NBgwZRunRp7O3t6dKli2GZwjvi4+Pp2LEjtra2lClThvfff5+cnByjOps2baJ+/fpYWVlRpUoVFixY8KS/3mOZNWsWdevWxdHREUdHR4KCgli7dq1he0ntlwf5/PPPUalUDB8+3FBWkvto4sSJqFQqo1f16tUN24tV35h1+Yhn1OLFixWNRqP89NNPytGjR5V+/fopzs7OSnJysrlDK3Rr1qxRPvroI2XFihUKoKxcudJo++eff644OTkpq1atUg4dOqS8/PLLiq+vr3L79m1DnXbt2in+/v7Krl27lK1btypVqlQxrIajKIpy8+ZNxcPDQ+nRo4cSExOjLFq0SLGxsVF++OGHp/U1CyQ4OFiZP3++EhMTo0RHRysdOnRQKlSooKSlpRnqDBgwQPH29laioqKUffv2Kc8995zSpEkTw/acnByldu3aSps2bZSDBw8qa9asUdzc3AwrEymKopw5c0axtbVVRowYoRw7dkyZPn26YmFhoaxbt+6pfl9TrF69WomIiFD+/fdfJTY2Vvnwww+VUqVKKTExMYqilNx+ycuePXuUihUrKnXr1jWsWqUoJbuPJkyYoNSqVUtJTEw0vC5fvmzYXpz6RpJwATRu3FgZNGiQ4bNWq1W8vLyU8PBwM0b15N2fhHU6nVK2bFnlyy+/NJTduHFDsbKyUhYtWqQoiqIcO3ZMAZS9e/ca6qxdu1ZRqVSGZfG+//57xcXFRcnMzDTUGT16tNHSe8+CS5cuKYCyefNmRVH0fVGqVCll2bJlhjrHjx9XAGXnzp2Kouj/yFGr1UpSUpKhzqxZsxRHR0dDf4waNUqpVauW0bG6deumBAcHP+mvVKhcXFyUefPmSb/cIzU1VfHz81MiIyONlo4s6X00YcIExd/fP89txa1v5HK0ibKysti/fz9t2rQxlKnVatq0acPOnTvNGNnTFxcXR1JSklFfODk5ERgYaOiLnTt34uzsTMOGDQ112rRpg1qtZvfu3YY6zZs3R6PRGOoEBwcTGxvL9evXn9K3eXw3b94E7i5VuH//frKzs436p3r16lSoUMGof+rUqWNYfhD03z0lJYWjR48a6tzbxp06z8rPm1arZfHixaSnpxMUFCT9co9BgwbRsWPHXN9D+ki/jKeXlxeVKlWiR48exMfHA8WvbyQJm+jKlStotVqjf1zQr9d6/0T9xd2d7/uwvkhKSqJMmTJG2y0tLXF1dTWqk1cb9x6jqNPpdAwfPpznn3/esM5vUlISGo0GZ2dno7r398+jvvuD6qSkpHD79u0n8XUKxZEjR7C3t8fKyooBAwawcuVKatasWeL75Y7Fixdz4MABwsPDc20r6X0UGBjIggULWLduHbNmzSIuLo5mzZqRmppa7PpGFnAQohAMGjSImJiYQl1q8FlXrVo1oqOjuXnzJsuXLycsLIzNmzebO6wi4fz58wwbNozIyEisra3NHU6R0759e8P7unXrEhgYiI+PD0uXLjUsvVlcyJmwidzc3LCwsMg1Ei85OZmyZcuaKSrzuPN9H9YXZcuW5dKlS0bbc3JyuHbtmlGdvNq49xhF2eDBg/nrr7/YuHEj5cuXN5SXLVuWrKwsbty4YVT//v551Hd/UB1HR8ci/QtJo9FQpUoVGjRoQHh4OP7+/nz77bclvl9Af0n10qVL1K9fH0tLSywtLdm8eTPfffcdlpaWeHh4lPg+upezszNVq1bl1KlTxe7nR5KwiTQaDQ0aNCAqKspQptPpiIqKIigoyIyRPX2+vr6ULVvWqC9SUlLYvXu3oS+CgoK4ceMG+/fvN9TZsGEDOp2OwMBAQ50tW7aQnZ1tqBMZGUm1atVwcXF5St/GdIqiMHjwYFauXMmGDRvw9fU12t6gQQNKlSpl1D+xsbHEx8cb9c+RI0eM/lCJjIzE0dGRmjVrGurc28adOs/az5tOpyMzM1P6Bf0ykkeOHCE6OtrwatiwIT169DC8L+l9dK+0tDROnz6Np6dn8fv5earDwIqJxYsXK1ZWVsqCBQuUY8eOKf3791ecnZ2NRuIVF6mpqcrBgweVgwcPKoAydepU5eDBg8q5c+cURdE/ouTs7Kz8+eefyuHDh5VOnTrl+YhSQECAsnv3bmXbtm2Kn5+f0SNKN27cUDw8PJS33npLiYmJURYvXqzY2toW+UeUBg4cqDg5OSmbNm0yepTi1q1bhjoDBgxQKlSooGzYsEHZt2+fEhQUpAQFBRm233mU4sUXX1Sio6OVdevWKe7u7nk+SvH+++8rx48fV2bOnFnkHzP54IMPlM2bNytxcXHK4cOHlQ8++EBRqVTK33//rShKye2Xh7l3dLSilOw+eu+995RNmzYpcXFxyvbt25U2bdoobm5uyqVLlxRFKV59I0m4gKZPn65UqFBB0Wg0SuPGjZVdu3aZO6QnYuPGjQqQ6xUWFqYoiv4xpXHjxikeHh6KlZWV0rp1ayU2NtaojatXryrdu3dX7O3tFUdHR6V3795KamqqUZ1Dhw4pTZs2VaysrJRy5copn3/++dP6igWWV78Ayvz58w11bt++rbzzzjuKi4uLYmtrq3Tu3FlJTEw0aufs2bNK+/btFRsbG8XNzU157733lOzsbKM6GzduVOrVq6doNBqlUqVKRscoivr06aP4+PgoGo1GcXd3V1q3bm1IwIpScvvlYe5PwiW5j7p166Z4enoqGo1GKVeunNKtWzfl1KlThu3FqW9kFSUhhBDCTOSesBBCCGEmkoSFEEIIM5EkLIQQQpiJJGEhhBDCTCQJCyGEEGYiSVgIIYQwE0nCjyEzM5OJEyeSmZlp7lCKJOmfB5O+eTjpn4eT/nmwZ61v5Dnhx5CSkoKTkxM3b97E0dHR3OEUOdI/DyZ983DSPw8n/fNgz1rfyJmwEEIIYSaShIUQQggzKXHrCefk5HDw4EE8PDxQqx/vb5DU1FQAEhISSElJKYzwihXpnweTvnk46Z+Hk/55sKLQNzqdjuTkZAICArC0fHiaLXH3hPfu3Uvjxo3NHYYQQohibs+ePTRq1OihdUrcmbCHhweg7xxPT08zRyOEEKK4SUxMpHHjxoZ88zAlLgnfuQTt6elJ+fLlzRyNEEKI4io/tzxlYJYQQghhJmZNwlu2bCEkJAQvLy9UKhWrVq165D6bNm2ifv36WFlZUaVKFRYsWPDE4xRCCCGeBLMm4fT0dPz9/Zk5c2a+6sfFxdGxY0deeOEFoqOjGT58OG+//Tbr169/wpEKIYQQhc+s94Tbt29P+/bt811/9uzZ+Pr68vXXXwNQo0YNtm3bxjfffENwcHChxqbVasnOzi7UNoUoCjQazWM/nieEKBzP1MCsnTt30qZNG6Oy4OBghg8fXmjHUBSFpKQkbty4UWhtClGUqNVqfH190Wg05g5FPEBGtpZ9Z6+TrdWZO5QSx93BitrlnJ7a8Z6pJJyUlJRryLeHhwcpKSncvn0bGxubXPtkZmYaTeR950Huhx3jxo0blClTBltbW1QqVeEEL0QRoNPpuHjxIomJiVSoUEF+vougDSeSmbD6KOev3TZ3KCXSS3U9mfFG/ad2vGcqCRdEeHg4kyZNylddrVZrSMClS5d+wpEJYR7u7u5cvHiRnJwcSpUqZe5wxH8uXL/FpP8dI/JYMgBu9hq8nHOfWIgnq4Kr7VM93jOVhMuWLUtycrJRWXJyMo6OjnmeBQOMGTOGESNGGD4nJCRQs2bNPOveuQdsa/t0/xGEeJruXIbWarWShIuAzBwt87bGMX3DSTKydViqVfRt6svQ1n7YWT1Tv6JFATxT/8JBQUGsWbPGqCwyMpKgoKAH7mNlZYWVlZXhc37mEpVLdKI4k5/vomP7qSuM+zOGM5fTAQj0deXj0NpU9XAwc2TiaTFrEk5LS+PUqVOGz3FxcURHR+Pq6kqFChUYM2YMCQkJ/PLLLwAMGDCAGTNmMGrUKPr06cOGDRtYunQpERER5voKQghhsuSUDD7+6xh/HU4EwM3eirEda9Cpnpf8kVTCmPU5hX379hEQEEBAQAAAI0aMICAggPHjxwP6+Tfj4+MN9X19fYmIiCAyMhJ/f3++/vpr5s2bV+iPJwm9ihUrMm3atHzX37RpEyqVSkaWC/EAOVod87aeofXXm/nrcCJqFfRqUpGo91oQGlBOEnAJZNYz4ZYtW/KwRZzymg2rZcuWHDx48AlG9ex51H/cCRMmMHHiRJPb3bt3L3Z2dvmu36RJExITE3FyenrD+4V4Vuw9e41xq2I4kaR/QiOggjMfd6r9VB+HEUXPM3VPWOQtMTHR8H7JkiWMHz+e2NhYQ5m9vb3hvaIoaLXaR65xCfpRtKbQaDSULVvWpH2Ki6ysLHnuVuTpSlom4WtO8MeBCwC42Jbig/bVea2BN2q1nPmWdDJtTjFQtmxZw8vJyQmVSmX4fOLECRwcHFi7di0NGjTAysqKbdu2cfr0aTp16oSHhwf29vY0atSIf/75x6jd+y9Hq1Qq5s2bR+fOnbG1tcXPz4/Vq1cbtt9/OXrBggU4Ozuzfv16atSogb29Pe3atTP6oyEnJ4ehQ4fi7OxM6dKlGT16NGFhYYSGhj7w+169epXu3btTrlw5bG1tqVOnDosWLTKqo9PpmDJlClWqVMHKyooKFSrw6aefGrZfuHCB7t274+rqip2dHQ0bNmT37t0A9OrVK9fxhw8fTsuWLQ2fW7ZsyeDBgxk+fDhubm6GWyJTp06lTp062NnZ4e3tzTvvvENaWppRW9u3b6dly5bY2tri4uJCcHAw169f55dffqF06dJGz7UDhIaG8tZbbz2wP0TRpNUp/LrrHK2+2mRIwN0be7PhvZZ0a1RBErAAJAk/kqIo3MrKMcvrYZfqTfXBBx/w+eefc/z4cerWrUtaWhodOnQgKiqKgwcP0q5dO0JCQozuwedl0qRJdO3alcOHD9OhQwd69OjBtWvXHlj/1q1bfPXVV/z6669s2bKF+Ph4Ro4cadj+xRdf8PvvvzN//ny2b99OSkrKIxfyyMjIoEGDBkRERBATE0P//v1566232LNnj6HOmDFj+Pzzzxk3bhzHjh1j4cKFhole0tLSaNGiBQkJCaxevZpDhw4xatQodDrTZif6+eef0Wg0bN++ndmzZwP62ai+++47jh49ys8//8yGDRsYNWqUYZ/o6Ghat25NzZo12blzJ9u2bSMkJAStVstrr72GVqs1+sPm0qVLRERE0KdPH5NiE+Z16PwNOn+/nXGrYkjJyKGWlyMr3mlC+Ct1cbGTKybiLrkc/Qi3s7XUHG+eBSKOTQ7GVlM4/0STJ0+mbdu2hs+urq74+/sbPn/88cesXLmS1atXM3jw4Ae206tXL7p37w7AZ599xnfffceePXto165dnvWzs7OZPXs2lStXBmDw4MFMnjzZsH369OmMGTOGzp07AzBjxoxcj6Hdr1y5ckaJfMiQIaxfv56lS5fSuHFjUlNT+fbbb5kxYwZhYWEAVK5cmaZNmwKwcOFCLl++zN69e3F1dQWgSpUqDz1mXvz8/JgyZYpR2b1TqFasWJFPPvmEAQMG8P333wMwZcoUGjZsaPgMUKtWLcP7N954g/nz5/Paa68B8Ntvv1GhQgWjs3BRdN24lcWX62NZuCceRQEHa0tGvliNN5/zwULOfEUeJAmXEA0bNjT6nJaWxsSJE4mIiCAxMZGcnBxu3779yDPhunXrGt7b2dnh6OjIpUuXHljf1tbWkIABPD09DfVv3rxJcnIyjRs3Nmy3sLCgQYMGDz0r1Wq1fPbZZyxdupSEhASysrLIzMw0TLJy/PhxMjMzad26dZ77R0dHExAQYEjABdWgQYNcZf/88w/h4eGcOHGClJQUcnJyyMjI4NatW9ja2hIdHW1IsHnp168fjRo1IiEhgXLlyrFgwQJ69eolo2aLOJ1OYfmBC3y+9gTX0rMAeCWgHGM61MDdweoRe4uSTJLwI9iUsuDYZPM8AmVTyqLQ2rp/lPPIkSOJjIzkq6++okqVKtjY2PDqq6+SlZX10Hbun2FJpVI9NGHmVf9xL7N/+eWXfPvtt0ybNs1w/3X48OGG2B80e9odj9quVqtzxZjXilr39+nZs2d56aWXGDhwIJ9++imurq5s27aNvn37kpWVha2t7SOPHRAQgL+/P7/88gsvvvgiR48elefgi7hjF1MY92cM+89dB6Cqhz0fd6pNYCWZ+lY8miThR1CpVIV2Sbgo2b59O7169TJcBk5LS+Ps2bNPNQYnJyc8PDzYu3cvzZs3B/RnuQcOHKBevXoP3G/79u106tSJN998E9APwvr3338N05H6+flhY2NDVFQUb7/9dq7969aty7x587h27VqeZ8Pu7u7ExMQYlUVHRz9yisf9+/ej0+n4+uuvDUsFLl26NNexo6KiHjqf+dtvv820adNISEigTZs2eHt7P/S4wjxSM7L5JvIkP+88i1anYKuxYHgbP3o/70spi8ccbqPTwfU40OaxnKpTObD6b0at2zcgNQk0tuBc4W6dy/+CYuIKTA4eYOOif5+ZBjcvgKUVuPrerXP1dN4xPYydO9j99wdJ9m24fg7UluB2zy2g62chO8O0dm1c9DGDPqarp0GlAvdqd+vcOA9Z6flv09oJHD1Ni+MxFb/sIvLFz8+PFStWEBISgkqlYty4cSYPTCoMQ4YMITw8nCpVqlC9enWmT5/O9evXH3r51c/Pj+XLl7Njxw5cXFyYOnUqycnJhiRsbW3N6NGjGTVqFBqNhueff57Lly9z9OhR+vbtS/fu3fnss88IDQ0lPDwcT09PDh48iJeXF0FBQbRq1Yovv/ySX375haCgIH777TdiYmIMk8o8SJUqVcjOzmb69OmEhIQYDdi6Y8yYMdSpU4d33nmHAQMGoNFo2LhxI6+99hpubm6A/r7wyJEjmTt3rmG2OFF0KIrC6kMX+TTiOJdS9SPZO9bxZOxLNfB0eswFF7Iz4MhS2DEDrsTmXaf7Yqj23zrs/66Dlf8HlVvDWyvu1pn7AmSl5b3/g7w8Her31L+P3wW/dwFPf/i/LXfr/PaKPmGaovUEaPbf/P2XT8CcluBYDkYcu1tneV9I2Gdau0GDIfi/Jx7SkuH7QLCwgnH33B5bM1LfR/kV8CZ0mmlaHI9JknAJNXXqVPr06UOTJk1wc3Nj9OjR+ZpXu7CNHj2apKQkevbsiYWFBf379yc4OBgLiwdfih87dixnzpwhODgYW1tb+vfvT2hoKDdv3jTUGTduHJaWlowfP56LFy/i6enJgAEDAP3zzH///TfvvfceHTp0ICcnh5o1azJzpv4/X3BwMOPGjWPUqFFkZGTQp08fevbsyZEjRx76Xfz9/Zk6dSpffPEFY8aMoXnz5oSHh9OzZ09DnapVq/L333/z4Ycf0rhxY2xsbAgMDDQMdgP9FYIuXboQERHx0Ee1xNN36lIq4/88yo7TVwHwdbNj0su1aF7VtGfqc7l1Dfb9CLvnQPp/ScTCCqzsc9e1uOeKjIUGbEuDtaNxHRtX/VmsKSyt72nX8r9275tIxMYFMh++HGwupe75w0T9X7t3zrjvsHbSl5vU7j0L7ajU+v0t7vvOVg6mtavJo7+fMJVSmM/BPAMuXLiAt7c358+fp3z58kbbMjIyiIuLw9fXF2tr6we0IJ4knU5HjRo16Nq1Kx9//LG5wzGb1q1bU6tWLb777rtCb1t+zk13KyuH6RtOMW/rGbK1ClaWaga/UIX+LSphZfkYYzeun4Wd38PBXyH7lr7MsTw8N1B/Vnp/chXPhIflmfvJmbAwq3PnzvH333/TokULMjMzmTFjBnFxcbzxxhvmDs0srl+/zqZNm9i0aZPRY0zCPBRFYf3RZD7+6xgJN24D0KZGGSaE1MK7MNad3TED9s7Vv/eoA88PhVqdjc92RbEmSViYlVqtZsGCBYwcORJFUahduzb//PMPNWrUMHdoZhEQEMD169f54osvqFat2qN3EE/MuavpTFh9lE2xlwEo52zDxJdr0bamR8Ea1OngVKT+fmjZ2vqyoHf0A7CCBkOllvqBRaJEkSQszMrb25vt27ebO4wi42mPUBe5ZWRrmb35NN9vOk1Wjo5SFir+r3llBr1QBRvNY1x63vAxbJsKNV6Gbr/qy1wrwZt/FE7g4pkkSVgIIf6zMfYSE1cf5dxV/f3ZplXcmNSpFpXdCzBg59Y1yMm8+8hL3a6w90d94lUUOesVgCRhIYTg4o3bTP7fMdYdTQLAw9GKcS/VpGMdT9NnK7t+FnbNggO/Qo0QeOUHfXmZGjAy1ni0sCjxJAkLIUqsrBwdP26L47uok9zO1mKhVtHn+YoMa1MVeysTfz0mHIAd0+HYqrsTZVyJBW2O/pEfkAQscpEkLIQokXacvsL4P49y6pJ+UovGFV2ZHFqL6mVNeCzozmCrHdPh7Na75ZVbQZOhMthKPJIkYSFEiXIpJYNP1xznz+iLALjZaxjTvgav1C+X/0vPOZlweCnsnKGfBQr0E1HUfhWaDLk7+lmIR5AkLIQoEXK0On7ZeY5vIv8lNTMHlQrees6H916shpNNPp/LvX0d9v0Eu3/QT5UIYOUIDXpB4AD9vM5CmOAxZxkXxUnLli1zrYc7bdq0h+6jUqlYtWrVYx+7sNoRIi/7z10nZMZ2Jv91jNTMHPy9nVk9qCmTO9XOfwIGWNEfoibrE7BjOXjxE3g3Bl78WBKwKBA5Ey4GQkJCyM7OZt263BOVb926lebNm3Po0CGjtYDzY+/evbmW63tcEydOZNWqVURHRxuVJyYm4uLikvdOQhTQ1bRMvlh3gqX7LgDgZFOK0e2q83ojb9TqfFx6vngQnLzBTr+4Bo36QUqi/pJz7VdkZivx2CQJFwN9+/alS5cuXLhwIdc8pfPnz6dhw4YmJ2DQL+n3tJQtW/apHasoycrKQqPRmDuMYkenU1i0N54p62K5eVu/9F63ht6Mbl8dV7t89veaUbDnB2gxGl74UF/m11b/ksFWopDI5ehi4KWXXsLd3Z0FCxYYlaelpbFs2TL69u3L1atX6d69O+XKlcPW1pY6deqwaNGih7Z7/+XokydP0rx5c6ytralZsyaRkZG59hk9ejRVq1bF1taWSpUqMW7cOLKz9b8EFyxYwKRJkzh06BAqlQqVSmWI+f7L0UeOHKFVq1bY2NhQunRp+vfvT1ra3aXZevXqRWhoKF999RWenp6ULl2aQYMGGY6Vl9OnT9OpUyc8PDywt7enUaNG/PPPP0Z1MjMzGT16NN7e3lhZWVGlShV+/PFHw/ajR4/y0ksv4ejoiIODA82aNeP06dNA7sv5AKGhofTq1cuoTz/++GN69uyJo6Mj/fv3f2S/3fG///2PRo0aYW1tjZubm2Et6MmTJ1O7du6BQPXq1WPcuHEP7I/i6siFm3SetYOPVsZw83Y2NTwd+WNgEF+8WvfhCTgnE7Ju3f3sE6QfbJVxd3UuVCpJwKJQyZlwfpmyMPQdFlZ3nw/U5oA2U7/k1r3PCj6oXU3+LwNbWlrSs2dPFixYwEcffWQY4bls2TK0Wi3du3cnLS2NBg0aMHr0aBwdHYmIiOCtt96icuXKNG7c+JHH0Ol0vPLKK3h4eLB7925u3ryZK+EAODg4sGDBAry8vDhy5Aj9+vXDwcGBUaNG0a1bN2JiYli3bp0h+Tk5OeVqIz09neDgYIKCgti7dy+XLl3i7bffZvDgwUZ/aGzcuBFPT082btzIqVOn6NatG/Xq1aNfv355foe0tDQ6dOjAp59+ipWVFb/88gshISHExsZSoYJ+QfSePXuyc+dOvvvuO/z9/YmLi+PKlSsAJCQk0Lx5c1q2bMmGDRtwdHRk+/bt5OTkPLL/7vXVV18xfvx4JkyYkK9+A4iIiKBz58589NFH/PLLL2RlZbFmzRoA+vTpw6RJk9i7dy+NGjUC4ODBgxw+fJgVK1bkDqCYunkrm6/+juW33edQFLC3suS9F6vy1nM+WFo85Hzj3sFWzw2Epu/qy2u8DMMOy71e8WQpJcz58+cVQDl//nyubbdv31aOHTum3L59O/eOExxNf8WsuLt/zAp92U8djNv9wjfvfU10/PhxBVA2btxoKGvWrJny5ptvPnCfjh07Ku+9957hc4sWLZRhw4YZPvv4+CjffPONoiiKsn79esXS0lJJSEgwbF+7dq0CKCtXrnzgMb788kulQYMGhs8TJkxQ/P39c9W7t505c+YoLi4uSlpammF7RESEolarlaSkJEVRFCUsLEzx8fFRcnJyDHVee+01pVu3bg+MJS+1atVSpk+friiKosTGxiqAEhkZmWfdMWPGKL6+vkpWVlae2+/vP0VRlE6dOilhYWGGzz4+PkpoaOgj47q/34KCgpQePXo8sH779u2VgQMHGj4PGTJEadmyZZ51H/pz/gzS6XTK8n3nlfqT/1Z8Rv+l+Iz+Sxm66ICSfPMR3+/aWUVZM1pRPvG8+//uh5aKotM9ncBFsfWwPHM/ORMuJqpXr06TJk346aefaNmyJadOnWLr1q1MnjwZAK1Wy2effcbSpUtJSEggKyuLzMxMbG3ztxzb8ePH8fb2xsvLy1AWFBSUq96SJUv47rvvOH36NGlpaeTk5ODoaNqaqMePH8ff399oUNjzzz+PTqcjNjYWDw/9Kja1atXCwuLuhPqenp4cOXLkge2mpaUxceJEIiIiSExMJCcnh9u3bxMfHw9AdHQ0FhYWtGjRIs/9o6OjadasGaVKPd5gnIYNG+Yqe1S/RUdHP/AMH6Bfv3706dOHqVOnolarWbhwId98881jxfksOJGUwvhVR9lz9hoAVcrYM7lTLZpUdnvwThcP6ifXOLoKFK2+rEyt/5YRfEUuN4unSpJwfn140fR9LKzuvq8eom9Ddd9lseEPThqm6tu3L0OGDGHmzJnMnz+fypUrGxLKl19+ybfffsu0adOoU6cOdnZ2DB8+nKysrEI7/s6dO+nRoweTJk0iODgYJycnFi9ezNdff11ox7jX/clQpVKh0+keWH/kyJFERkby1VdfUaVKFWxsbHj11VcNfWBj8/ApBR+1Xa1WoyiKUVle96jvH3Gen3571LFDQkKwsrJi5cqVaDQasrOzefXVVx+6z7MsLTOHaZH/Mn/HWbQ6BZtSFgxr40ef533RWOZx6Vmng1P/wI7vjGe2qtRSP7NV5VaSfIVZSBLOLxPu0ebJwvLu/eHCbPceXbt2ZdiwYSxcuJBffvmFgQMHGu4Pb9++nU6dOvHmm28C+nu8//77LzVr1sxX2zVq1OD8+fMkJibi6alfFWbXrl1GdXbs2IGPjw8fffSRoezcuXNGdTQaDVqt9pHHWrBgAenp6YaEtX37dtRq9WOtsbt9+3Z69eplGNCUlpZmtHRgnTp10Ol0bN68mTZt2uTav27duvz8889kZ2fneTbs7u5OYmKi4bNWqyUmJoYXXnjhoXHlp9/q1q1LVFQUvXv3zrMNS0tLwsLCmD9/PhqNhtdff/2RiftZpCgKEUcS+fivYySnZALQrlZZxoXUpJxzHt83JxOOLNOf+d6Z2UplAbW76B8z8jT9qQEhCpOMji5G7O3t6datG2PGjCExMdFoVK6fnx+RkZHs2LGD48eP83//938kJyfnu+02bdpQtWpVwsLCOHToEFu3bjVKGneOER8fz+LFizl9+jTfffcdK1euNKpTsWJF4uLiiI6O5sqVK2RmZuY6Vo8ePbC2tiYsLIyYmBg2btzIkCFDeOuttwyXogvCz8+PFStWEB0dzaFDh3jjjTeMzpwrVqxIWFgYffr0YdWqVcTFxbFp0yaWLl0KwODBg0lJSeH1119n3759nDx5kl9//ZXY2FgAWrVqRUREBBEREZw4cYKBAwdy48aNfMX1qH6bMGECixYtYsKECRw/fpwjR47wxRdfGNV5++232bBhA+vWraNPnz4F7qei6vTlNN76cQ+DFx4kOSUTn9K2LOjdiNlvNcg7AQP81A7+HKRPwBp7CBoMww5Bl7mSgEWRIEm4mOnbty/Xr18nODjY6P7t2LFjqV+/PsHBwbRs2ZKyZcsSGhqa73bVajUrV67k9u3bNG7cmLfffptPP/3UqM7LL7/Mu+++y+DBg6lXrx47duzI9YhMly5daNeuHS+88ALu7u55PiZla2vL+vXruXbtGo0aNeLVV1+ldevWzJgxw7TOuM/UqVNxcXGhSZMmhISEEBwcTP369Y3qzJo1i1dffZV33nmH6tWr069fP9LT9SPYS5cuzYYNG0hLS6NFixY0aNCAuXPnGs6K+/TpQ1hYGD179qRFixZUqlTpkWfBkL9+a9myJcuWLWP16tXUq1ePVq1asWfPHqM6fn5+NGnShOrVqxMYGPg4XVWk3M7S8tX6WNpN28K2U1fQWKoZ3saP9cOb07JaGePKN+L1TyLcUSsUHDyh7WR49ygEfwrO3k81fiEeRqXcfxOrmLtw4QLe3t6cP38+18QWGRkZxMXF4evri7W1tZkiFKJgFEXBz8+Pd955hxEjRjyw3rP0cx55LJmJq4+ScOM2AC9Uc2fiy7XwKZ3HbZw178PeH/VnubW76Muyb+svP1vKhCji6XlYnrmf3BMWohi4fPkyixcvJikp6YH3jZ8l56/dYuLqo0SduARAOWcbxofU5MWaHndXOrpz/nDns21p/Wjn83vuJmFZv1cUcZKEhSgGypQpg5ubG3PmzHmm5+DOzNHyw+YzzNx4iswcHaUsVLzdrBJDWlXBVvPfr6ucLP1gq50zoPUEqNZOX964P1RrD57+5vsCQphIkrAQxUBxuKu05d/LTFh9lLgr+nvwTSqXZnKn2lQpY6+vcPsG7J+vn9kq9b9R6Hvn3k3Ctq76lxDPEEnCQgizSrx5m4//OsaaI0kAlHGwYuxLNQmp66m/9HwjHnbNhgM/Q9Z/84c7eOrX723Qy3yBC1EIJAkLIcwiW6tj/vY4pv1zkltZWizUKsKCKvJuWz8crEtB4iH9870xK4xntmoyRH/PVwZbiWJAknAeHjbrkhDPuqJw6XrXmauM/zOGf5P1Z7YNfVyY3Kk2NT0d4FSUfmaruM13d6jUUp98K7eWma1EsSJJ+B4ajQa1Ws3Fixdxd3dHo9HcHYkpRDGgKAqXL19GpVI99hzYBXEpNYPwNSdYeTABAFc7DWPaV6dL/fKoFS3MaQmJ0frKhpmtBstgK1FsSRK+h1qtxtfXl8TERC5eLMBc0UI8A1QqFeXLlzda/OJJ0+oUftt1jq/Wx5KamYNKBW80rsD7L5TH2fnOaG5LKFMTrp7S3+sNHCATa4hiT5LwfTQaDRUqVCAnJ+eRcxwL8SwqVarUU03AB+KvM25VDEcvpgBQp5wTn3Sqhf+Jr+H7BdBnHZStra/cZgK0Cwcb56cWnxDmJEk4D3cu1Znjcp0QxcX19CymrD/Boj3nAXC0tuT9dtV5o3EFLNQq2BUPWakQs/xuEnYoa8aIhXj6JAkLIQqVTqewdN95vlh3guu3sgGFD6sl0ov/ofH7BtT/jbNo8QEE9IQqrc0arxDmJElYCFFoYhJuMu7PGA7G36AUOQx2PcA7mrXYntOvNMXOmfDSVP17j5r6lxAlmCRhIcRjS8nIZurf//LLzrPYK+kM0WxkgE0kdrcuwy30ywjWD4PnBpo7VCGKFEnCQogCUxSFVdEJfBpxAk1aAmMs1/GmZhM2uluQifHMVjLYSohcJAkLIQrk3+RUxq2KIe3sAT6yjOBl651YoAMd+keNmgyB2q/KzFZCPITa3AHMnDmTihUrYm1tTWBgYK6Fyu+VnZ3N5MmTqVy5MtbW1vj7+7Nu3bqnGK0QIj0zh/A1x+n67XqGXHiPCKsP6WyxXZ+AfVtAjz9g4A6o94YkYCEewaxnwkuWLGHEiBHMnj2bwMBApk2bRnBwMLGxsZQpUyZX/bFjx/Lbb78xd+5cqlevzvr16+ncuTM7duwgICDADN9AiJJDURTWHknk44jjJN7MAKwp75CDkmWBqvYrEDQYvOqZO0whnikqxYwTyQYGBtKoUSNmzJgB6Ods9vb2ZsiQIXzwwQe56nt5efHRRx8xaNAgQ1mXLl2wsbHht99+y9cxL1y4gLe3N+fPn6d8+fKF80WEKObikq+za+EnNLy+li5ZE3FydWPSy7Vo5ZioXz7QuYK5QxSiyDAlz5h8JlyxYkX69OlDr169qFCh4P/xsrKy2L9/P2PGjDGUqdVq2rRpw86dO/PcJzMzE2tra6MyGxsbtm3b9sDjZGZmkpmZaficmppa4JiFKFFysriYpuWnbXH8svMs/7NYh586gW9rHCPojXFYl7IAPMwdpRDPNJPvCQ8fPpwVK1ZQqVIl2rZty+LFi42SXH5duXIFrVaLh4fxf2IPDw+SkpLy3Cc4OJipU6dy8uRJdDodkZGRrFixgsTExAceJzw8HCcnJ8OrZk15LlGIB0pJhH3zSf3pFTI+8+GlKf9j3rY4srQKEWX6c7n1N7zQY8x/CVgI8bgKlISjo6PZs2cPNWrUYMiQIXh6ejJ48GAOHDjwJGI0+Pbbb/Hz86N69epoNBoGDx5M7969Uasf/DXGjBnDzZs3Da9jx4490RiFeKYoCiQdgc1TUOa8AFOrw1/DcYiPwlp3i8YcJahSaeb3bsS7g4bi3qwPWFqZO2ohio0CD8yqX78+9evX5+uvv+b7779n9OjRzJo1izp16jB06FB69+790GUA3dzcsLCwIDk52ag8OTmZsmXznj/W3d2dVatWkZGRwdWrV/Hy8uKDDz6gUqVKDzyOlZUVVlZ3f2mkpKSY+E2FKGZyMuHsNohdq3+lXADgzv/WaF1lonQNyKzSjkGtW1PH29lsoQpR3BU4CWdnZ7Ny5Urmz59PZGQkzz33HH379uXChQt8+OGH/PPPPyxcuPCB+2s0Gho0aEBUVBShoaGAfmBWVFQUgwcPfuixra2tKVeuHNnZ2fzxxx907dq1oF9DiJLj6Er961QUZKUZijPQsFVbh3909dlp0YA2jfzp/XxFvF1tzRisECWDyUn4wIEDzJ8/n0WLFqFWq+nZsyfffPMN1atXN9Tp3LkzjRo1emRbI0aMICwsjIYNG9K4cWOmTZtGeno6vXv3BqBnz56UK1eO8PBwAHbv3k1CQgL16tUjISGBiRMnotPpGDVqlKlfQ4ji79oZcL3nKtGR5XDiLwBSS7mxLsuftdkB7NDVwsHBkd7PV+TDxj442crqYUI8LSYn4UaNGtG2bVtmzZpFaGhonsv9+fr68vrrrz+yrW7dunH58mXGjx9PUlIS9erVY926dYbBWvHx8Ub3ezMyMhg7dixnzpzB3t6eDh068Ouvv+Ls7Gzq1xCi+NLmwOymcPk4DN4PblUAiPfpwonLrsxKqkp0RkUU1PiVsWdy80p0queFlaUMthLiaTP5OeFz587h4+PzpOJ54uQ5YVGsZKTAqX/g0nFo9dHd8p9fhnM7UF6Zy1ZNU+ZuPcPWk1cMm4MqlaZ/80q0qOqOWv3gsRtCCNM90eeEL126RFJSEoGBgUblu3fvxsLCgoYNG5rapBDCFDfiIXYdxK7RD7DSZevLG/UFB/2gxqz237AuLpvv/7nEiST9VLAWahUd6njSr5kvdcs7myl4IcS9TE7CgwYNYtSoUbmScEJCAl988QW7d+8utOCEEIBOBxcPwr//jWZOjjHeXroKVGsPikJKRjaL98Tz07azJKVkAGCrsaBbI2/6PO8rg62EKGJMTsLHjh2jfv36ucoDAgLkGVwhCkvWLYjbrE+6/66DtHse5VOpoUIQVG2nT75ufly8cZsF286ycPdh0jJzAHB3sKJXk4q8GSiDrYQoqkxOwlZWViQnJ+d6NjcxMRFLS1kZUYjHpigwo5Hh+V0ANA5QpTVU6wB+bfXzNQPHLqYwd0k0/zt0kRydfniHXxl7+slgKyGeCSZnzRdffJExY8bw559/4uTkBMCNGzf48MMPadu2baEHKESxlpECe36AC/uh+yJQqfSvik3h3Hb9mW619uDT1LAsoKIobDt5mTlbjAdbPVfJlf9rXlkGWwnxDDE5CX/11Vc0b94cHx8fw/KB0dHReHh48OuvvxZ6gEIUKzlZ+jPcO8/vWmhg61TIvgVJh8HTX1/e8WvQ2OkT8n+ytTr+d+gic7ac4USSfiEStQo61vWSwVZCPKNMTsLlypXj8OHD/P777xw6dAgbGxt69+5N9+7d83xmWIgS79Y1OBmpH1h1KgocvWDQfwMYS1lD85FgWxqcvO/uY2VveJuakc2iPfHM3372v3V8ZbCVEMVFgW7i2tnZ0b9//8KORYji48qpu6OZ43eBor277Za1PjH/d1+XZu/l2UTizdvM336WRbvjSb1vsFWPwAo422qe9LcQQjxhBR5JdezYMeLj48nKyjIqf/nllx87KCGeOdocuLDn7qIIV08aby9T67/7ux3AKwAesvLXsYspzNt6htX3DLaqUsae/s0q0SlABlsJUZyYnITPnDlD586dOXLkCCqVijsTbt1ZMUmr1T5sdyGKl8xUiBgJJ/+G29fulqtL6QdXVWuvf5TI5eGzzCmKwrZTV3INtgr0deX/WlSiZdUyMthKiGLI5CQ8bNgwfH19iYqKwtfXlz179nD16lXee+89vvrqqycRoxBFx43zcPUUVH5B/1ljD2e36hOwtTNUDdYn3sqtwdrxkc1la3X8dfgic7bEcTxRv8ymWsV/M1tVwl+WERSiWDM5Ce/cuZMNGzbg5uaGWq1GrVbTtGlTwsPDGTp0KAcPHnwScQphfhf2w7xWYOMK758CtYV+9HK7z/UDq7wDwSJ//6VSM7JZvOc8P22PMwy2simlH2zVt6kMthKipDA5CWu1WhwcHABwc3Pj4sWLVKtWDR8fH2JjYws9QCGeuuzbcGazfmCVgye0/EBf7ukPtm7g5gfplw3zNFMz/+MgEm/eZsH2syy8Z7CVm70VvZ+XwVZClEQmJ+HatWtz6NAhfH19CQwMZMqUKWg0GubMmZNrFi0hnhlpl/TTQ8auhdMbIee2vtzJG1qM1p/xWljC8COgMf0s9XhiCnO3nmF19N3BVpXd7ejfvBKd6pXDupQMthKiJDI5CY8dO5b09HQAJk+ezEsvvUSzZs0oXbo0S5YsKfQAhXgiFAUuHbs7mjlhP3DPqp6O5e/OVqUodyfNMCEBK4rC9lNXmbP1DFv+vWwoD/R1pX/zSrxQTQZbCVHSmZyEg4ODDe+rVKnCiRMnuHbtGi4uLoYR0kIUSTlZ+qkg//1vGcAb8cbbvQL0jxBVaw8etY1mqzJFtlZHxOFE5mw5w7F7Blu1r+NJfxlsJYS4h0lJODs7GxsbG6Kjo6ldu7ah3NXVtdADE6JQ6HR3n8m9eR5+Db27zdIafFvcfYzI0fOxDpWakc2Svef5aVscF2WwlRAiH0xKwqVKlaJChQryLLAo+s7vgajJYOcGry3Ql5WurF8IwbWi/oy3Ukv9/MyPKelmBvO3x8lgKyGEyUy+HP3RRx/x4Ycf8uuvv8oZsCgadFq4sFf/zG7Z/67QWJTSP79byk5/Gfq/FYjoHVFohz2RlMKcLTLYSghRcCYn4RkzZnDq1Cm8vLzw8fHBzs74TOLAgQOFFpwQD5SZCqc3QOw6OLkebl0F/zeg8yz9ds960HGqfg1ey8I7E5XBVkKIwmRyEg4NDX0CYQiRT8f+hAO/QNwW0N4zb7m1E1g53P2sUkGjvoV22IcNturXrBL1ZLCVEKIATE7CEyZMeBJxCPFw2bdhzftw8J41q118745mrvCc/hJ0IXvYYKs+z/tSobQMthJCFFyBV1ES4qm5cgqWhUFyDKCCJkMg4E1wq1rgx4geJelmBvN3/DfYKuPuYKteTXzoEeiDi50MthJCPD6Tk7BarX7o88AycloUqpgVsHooZKWCnTt0macf1fyEnEhKYe6WOFYfSiBbe3ewVb9mlQgNkMFWQojCZXISXrlypdHn7OxsDh48yM8//8ykSZMKLTAh2DUb1o3Wv/d5Hrr8+NjP8uZFURR2nL7KnC1n2HzPYKvGvq70b1aJVtVlsJUQ4skwOQl36tQpV9mrr75KrVq1WLJkCX37Ft5gGFHC1XgJtkyB+j3hhbH5XqEov7K1OtYc0Q+2OnrxnsFWtT15u5kvARVcCvV4Qghxv0L7rfbcc8/Rv3//wmpOlFSXY8G9mv69U3kYvA9sC/d59LTMHBbviWf+9rMk3NAv1GBTyoKuDcvTp6kvPqUffwIPIYTIj0JJwrdv3+a7776jXLlyhdGcKIkUBSLHw47p8PpCqN5BX16ICTg5JYOftt8/2EpDWFBF3nxOBlsJIZ4+k5Pw/Qs1KIpCamoqtra2/Pbbb4UanChBVCrQ5QAKXDx4NwkXgtikVOZuPcOf0XcHW1X6b7BVZxlsJYQwI5OT8DfffGOUhNVqNe7u7gQGBuLiIvfQhIm0OXfv9baZBH5toXKrx25WURR2nr7KD/cPtqqon9lKBlsJIYoCk5Nwr169nkAYosTRaWFTOJzbCT3/1CdiS81jJ+A7g63mbj1DTMLdwVbtapelX7NKMthKCFGkmJyE58+fj729Pa+99ppR+bJly7h16xZhYWGFFpwoplKT4Y+++gUWAP5dCzVCHqvJvAZbWZdS062htwy2EkIUWSYn4fDwcH744Ydc5WXKlKF///6ShMXDxW2B5X0h/ZJ+haOQbx8rASenZDB/+1l+331OBlsJIZ45Jifh+Ph4fH19c5X7+PgQHx9fKEGJYking21fw8bPQNGBew3o+gu4Vy1QczLYSghRHJichMuUKcPhw4epWLGiUfmhQ4coXbp0YcUlipP0q7CiH5yO0n+u1wM6fAUa0xc/2H/uOtM3nGRTrPFgq37NK9FaBlsJIZ4xJifh7t27M3ToUBwcHGjevDkAmzdvZtiwYbz++uuFHqB4xsXvhuW9ISUBLG2g41f6xRdMpNMpfL/pFFMj/0WnyGArIUTxYHIS/vjjjzl79iytW7fG0lK/u06no2fPnnz22WeFHqB4RikK7JwB/0zUP/9b2g+6/gwetUxu6lp6FsOXRLPlv0eNQut58W7bqjLYSgjxzDM5CWs0GpYsWcInn3xCdHQ0NjY21KlTBx8fnycRn3gW3b4Bq96B2Aj959pd9AOwrBxMbmrf2WsMXniQpJQMrEupmdypNl0behduvEIIYSYFnrbSz88PPz+/woxFFBdqC7gSCxYaaPc5NOxj8rq/iqIwb2scn687gVanUMndju971Kd6WccnFLQQQjx9JifhLl260LhxY0aPHm1UPmXKFPbu3cuyZcsKLTjxDFH0I5RRqfRnvF1/BW0WeNUzuambt7IZufwQkceSAXjZ34vPXqmDvVXhrqIkhBDmpjZ1hy1bttChQ+55fdu3b8+WLVsKJSjxjMlI0Q++2jXrbplHzQIl4MMXbtBx+lYijyWjsVDzSWhtvn29niRgIUSxZPJvtrS0NDSa3BMglCpVipSUlEIJSjxjjv8Pjq6E2HVQtyvYuZnchKIo/LrrHJ/8dZwsrY4KrrZ836M+tcs5PYGAhRCiaDD5TLhOnTosWbIkV/nixYupWbNmoQQlnjH13oDAgRC2ukAJODUjm8GLDjL+z6NkaXUE1/Lgf0OaSgIWQhR7Jp8Jjxs3jldeeYXTp0/TqpV+sv2oqCgWLlzI8uXLCz1AUQRlpcOmz6H5SLB20t8Hbv95gZo6djGFQQsPEHclHUu1ijEdatDn+YpGK3UJIURxZXISDgkJYdWqVXz22WcsX74cGxsb/P392bBhA66uhbcAuyiiLsfC0jC4fBxuxOuf/S0ARVFYuu884/88SmaODi8na2b0qE99mXhDCFGCmHw5GqBjx45s376d9PR0zpw5Q9euXRk5ciT+/v4mtzVz5kwqVqyItbU1gYGB7Nmz56H1p02bRrVq1bCxscHb25t3332XjIyMgnwNYarDS2HOC/oEbO8BjfsVqJlbWTm8t+wQo/84QmaOjhequRMxtJkkYCFEiVPgIadbtmzhxx9/5I8//sDLy4tXXnmFmTNnmtTGkiVLGDFiBLNnzyYwMJBp06YRHBxMbGwsZcqUyVV/4cKFfPDBB/z00080adKEf//9l169eqFSqZg6dWpBv4p4lOwMWDca9i/Qf/ZtAV3mgX3uf6NHOXUplYG/HeDkpTTUKhgZXI0BzSvLnM9CiBLJpCSclJTEggUL+PHHH0lJSaFr165kZmayatWqAg3Kmjp1Kv369aN3794AzJ49m4iICH766Sc++OCDXPV37NjB888/zxtvvAFAxYoV6d69O7t37zb52CKfrp6GZWGQdARQQYtR0GK0fkIOE606mMCHK49wK0tLGQcrvusewHOVZNEPIUTJle/L0SEhIVSrVo3Dhw8zbdo0Ll68yPTp0wt84KysLPbv30+bNm3uBqNW06ZNG3bu3JnnPk2aNGH//v2GS9ZnzpxhzZo1eT63LArBsT9hTkt9ArYtDW/+AS98aHICzsjWMmbFEYYvieZWlpbnq5QmYmgzScBCiBIv32fCa9euZejQoQwcOLBQpqu8cuUKWq0WDw8Po3IPDw9OnDiR5z5vvPEGV65coWnTpiiKQk5ODgMGDODDDz984HEyMzPJzMw0fE5NTX3s2Iu9nCyIHA+7/5t8o0IQvPoTOHqZ3NTZK+m88/sBjiWmoFLB0FZ+DG3th4VcfhZCiPyfCW/bto3U1FQaNGhAYGAgM2bM4MqVK08ytlw2bdrEZ599xvfff8+BAwdYsWIFERERfPzxxw/cJzw8HCcnJ8NLnmV+hBvxML/d3QT8/DAI+1+BEvDaI4m8NH0bxxJTKG2n4Zc+jXm3bVVJwEII8R+VotyZ9Dd/0tPTWbJkCT/99BN79uxBq9UydepU+vTpg4ND/lfJycrKwtbWluXLlxMaGmooDwsL48aNG/z555+59mnWrBnPPfccX375paHst99+o3///qSlpaFW5/6b4v4z4YSEBGrWrMn58+cpX758vuMtMZa8BcdXg7UzdJ4N1dqb3ERWjo7wtceZv/0sAI0qujC9e33KOlkXbqxCCFEEXbhwAW9v73zlGZMfUbKzs6NPnz5s27aNI0eO8N577/H5559TpkwZXn755Xy3o9FoaNCgAVFRUYYynU5HVFQUQUFBee5z69atXInWwkJ/f/JBf0tYWVnh6OhoeJnyh0KJ1OErqNYB/m9LgRLwheu3eO2HnYYEPKBFZRb1e04SsBBC5KFAzwnfUa1aNaZMmcKFCxdYtGiRyfuPGDGCuXPn8vPPP3P8+HEGDhxIenq6YbR0z549GTNmjKF+SEgIs2bNYvHixcTFxREZGcm4ceMICQkxJGNhopRE2P3D3c8OHtB9EbiYvj501PFkOn63jUPnb+BkU4ofwxryQfvqWFo81o+ZEEIUW4WyNI2FhQWhoaFGl5Xzo1u3bly+fJnx48eTlJREvXr1WLdunWGwVnx8vNGZ79ixY1GpVIwdO5aEhATc3d0JCQnh008/LYyvUfJk3IQfmkP6Jf3o5zqvFqiZHK2OL/+O5YfNZwDw93Zm5hsBlHexLcxohRCi2DH5nvCzzpRr9SVC1Mfw73r99JOlK5u8e9LNDIYuOsies9cA6P18Rca0r4HGUs5+hRAlkyl5RhZpLWnSLkNOBjh76z+3HKNfiKGUjclNbT15meGLo7manoW9lSVTXq1LhzqehRywEEIUX5KES5Kz22F5H3AoC33/BksrsLDUv0yg1Sl8G3WS6RtOoihQ09OR73vUp6Kb3RMKXAghiidJwiWBTgc7vtVfela0+uUH0y+Dk+mX4y+nZjJ8yUG2n7oKQPfGFZgQUhPrUjIwTgghTCVJuLi7dQ1W/h+c/Fv/ue7r8NJU0Jh+1rr7zFWGLDrIpdRMbDUWfNa5DqEB5Qo5YCGEKDkkCRdn5/fCsl6QcgEsraH9FKjfE1SmzVil0ynM3nKar9bHolPAr4w9s96sT5Uy8sy1EEI8DknCxZGiwK5ZEDkOdDngWgm6/gJl65jc1PX0LEYsjWZj7GUAXgkoxyeda2OrkR8dIYR4XPKbtLi5fQP+HAQn/tJ/rhkKL08Ha0eTmzoQf53Bvx/g4s0MrCzVTO5Ui64NvVGZeCYthBAib5KEi5OL0fq1f6+fBXUpCP4MGvcz+fKzoij8tP0s4WuOk6NT8HWzY+Yb9anpZXoiF0II8WCShIuLuK3wWxfQZoJTBei6AMo1MLmZm7ezGbX8EOuPJgPQsY4nn3epg4N1qUIOWAghhCTh4qJ8Q3DzAydvCP0ebF1NbiIm4Sbv/H6A+Gu3KGWhYtxLNXnrOR+5/CyEEE+IJOFn2bUz4FwR1Gr9jFdh/wMblwJdfv59dzyT/3eMLK2O8i42zHyjPv7ezk8kbCGEEHoywe+z6tAS+L4JbP36bpmtq8kJOC0zh2GLoxm7KoYsrY42NTyIGNJMErAQQjwFcib8rNLlQM5tOL9bPyOW2vS/p04kpfDO7wc4czkdC7WKD9pV5+1mvnL5WQghnhJJws8SnRbU/00PGdBDf+m5anCBEvCyfecZ92cMGdk6yjpaM+ONABpWNP0+shBCiIKTy9HPipg/4PsgSL96t6x6h7tJOZ9uZ2l5f9kh3l9+mIxsHc2ruhMxtKkkYCGEMAM5Ey7qcjJh/Yewd57+866Z0Hp8gZo6fTmNQb8f4ERSKmoVjGhblXdaVkGtlsvPQghhDpKEi7Jrcfq5nxOj9Z+bjdSv/1sAqw9dZMwfh0nP0uJmb8V33evRpLJboYUqhBDCdJKEi6rjf8GqdyDzJti4witzwK+tyc1kZGv5JOIYv+2KB+C5Sq581z2AMg7WhR2xEEIIE0kSLmq02fDPRNg5Q/+5fGN4bX6B1v6Nv3qLdxbuJyYhBYAhraowrLUflhYyFEAIIYoCScJFyc0LsKw3XNij/xw0GNpMBAvTp4xcfzSJkcsOkZqRg4ttKb7pVo+W1coUbrxCCCEeiyThouJkJKzoD7evgZWTfurJGi+Z3Ey2VscXa08wb1scAA18XJjePQAvZ5vCjlgIIcRjkiRsbjotbPz07sxXnvXgtQXg6mtyUwk3bjN44QEOxt8AoF8zX0a1q04pufwshBBFkiRhs1NB8lH920Zv65cftLQyuZWNsZd4d0k0N25l42htyVev+fNirbKFHKsQQojCJEnYXBRFP8+zWg2hs+DsVqjZyeRmcrQ6vvnnX2ZuPA1A3fJOzHyjPt6utoUdsRBCiEImSfhp0+n0l56vn4VOM/SJ2Na1QAn4UkoGQxYdZHfcNQB6BvnwUccaWFmaNouWEEII85Ak/LQlH4FNn4Gig3rdoWLTAjWz49QVhi4+yJW0LOw0FnzepS4h/l6FHKwQQognSZLw0+bpD20/1i++UIAErNMpzNh4im/++RdFgeplHfi+R30quds/gWCFEEI8SZKEnzRFgZ0zwe9FcK+qL2syuEBNXU3LZPiSaLaevAJAt4beTOpUC+tScvlZCCGeRZKEn6Tb12HlQPh3LRz8DfpvglIFmy5y79lrDFl4kKSUDKxLqfkktA6vNjB9Fi0hhBBFhyThJyVhv37xhRvxYKGBwP4FevRIp1OYu/UMU9bHotUpVHa34/seDahW1qHwYxZCCPFUSRIubIoCe+bqlx/UZYNLRXjtZ/CqZ3JTN25lMXLZIf45fgmATvW8+KxzHeys5J9NCCGKA/ltXpgyUmD1EDi2Sv+5Rgh0mgnWTiY3FX3+BoN+P0DCjdtoLNVMDKlF98beqFSy9q8QQhQXkoQLS9IRWBoG106D2hJe/AQCB+ifAzaBoij8vOMsn645TrZWwae0LTPfqE/tcqYnciGEEEWbJOHHpShw4BdYOwpyMsCxvH7uZ+9GJjeVkpHNB38cZs2RJADa1y7LF6/WxdHa9FWUhBBCFH2ShB9HVjr8NQIOL9Z/9nsROv+gnwHLREcv3mTQ7wc4e/UWpSxUfNihBr2aVJTLz0IIUYxJEn4cu2frE7DKAlqPgybD9HNBm0BRFBbvPc+E1UfJytFRztmGGW8EEFDB5QkFLYQQoqiQJPw4goZAwgF47h2o+LzJu6dn5jB2VQwrDyYA0Kp6GaZ29cfZVlPYkQohhCiCJAk/DksNvP57gXY9mZzKwN8PcOpSGhZqFe8HV6N/s0qo1XL5WQghSgpJwmaw4sAFPloZw+1sLR6OVkzvXp/GvqbfRxZCCPFskyT8FGVka5n0v6Ms2nMegKZV3Jj2ej3c7E2fSUsIIcSzT5LwUxJ3JZ13fj/A8cQUVCoY1tqPIa38sJDLz0IIUWJJEn4KIg4nMvqPw6Rl5lDaTsO3rwfQ1M/N3GEJIYQwM0nCT1BmjpbPIo7z885zADT2dWV69wA8HAu2kpIQQojiRZLwE3L+2i0GLzzAoQs3ARjYsjLvta2KpYVpzxELIYQoviQJPwGRx5J5b2k0KRk5ONmU4ptu/rSq7mHusIQQQhQxkoQLUbZWx1frY/lhyxkA6nk7M+ONAMq72Jo5MiGEEEVRkbg2OnPmTCpWrIi1tTWBgYHs2bPngXVbtmyJSqXK9erYseNTjDi3xJu36T5nlyEB93nel6X/FyQJWAghxAOZ/Ux4yZIljBgxgtmzZxMYGMi0adMIDg4mNjaWMmXK5Kq/YsUKsrKyDJ+vXr2Kv78/r7322tMM28iWfy8zfEk019KzcLCy5MvX6tKutqfZ4hFCCPFsMPuZ8NSpU+nXrx+9e/emZs2azJ49G1tbW3766ac867u6ulK2bFnDKzIyEltbW7MkYa1OYerfsYTN38O19CxqeTny19CmkoCFEELki1nPhLOysti/fz9jxowxlKnVatq0acPOnTvz1caPP/7I66+/jp2dXZ7bMzMzyczMNHxOTU19vKD/cyk1g2GLotl55ioAPQIrMO6lmliXsiiU9oUQQhR/Zj0TvnLlClqtFg8P45HDHh4eJCUlPXL/PXv2EBMTw9tvv/3AOuHh4Tg5ORleNWvWfOy4Ac5fu83es9ew1Vjw7ev1+LRzHUnAQgghTGL2y9GP48cff6ROnTo0btz4gXXGjBnDzZs3Da9jx44VyrEb+Lgw5dW6rB7clE71yhVKm0IIIUoWs16OdnNzw8LCguTkZKPy5ORkypYt+9B909PTWbx4MZMnT35oPSsrK6ys7i6QkJKSUvCA7/NK/fKF1pYQQoiSx6xnwhqNhgYNGhAVFWUo0+l0REVFERQU9NB9ly1bRmZmJm+++eaTDlMIIYR4Isz+iNKIESMICwujYcOGNG7cmGnTppGenk7v3r0B6NmzJ+XKlSM8PNxovx9//JHQ0FBKly5tjrCFEEKIx2b2JNytWzcuX77M+PHjSUpKol69eqxbt84wWCs+Ph612viEPTY2lm3btvH333+bI2QhhBCiUKgURVHMHcTTdOHCBby9vTl//jzly8s9XSGEEIXLlDzzTI+OFkIIIZ5lZr8c/bTpdDoAEhMTzRyJEEKI4uhOfrmTbx6mxCXhO49DPezZYiGEEOJxJScnU6FChYfWKXH3hHNycjh48CAeHh65BnyZKjU1lZo1a3Ls2DEcHBwKKcLiR/op/6Sv8k/6Kn+kn/KvsPpKp9ORnJxMQEAAlpYPP9ctcUm4MKWkpODk5MTNmzdxdHQ0dzhFlvRT/klf5Z/0Vf5IP+WfOfpKBmYJIYQQZiJJWAghhDATScKPwcrKigkTJhjNTS1yk37KP+mr/JO+yh/pp/wzR1/JPWEhhBDCTORMWAghhDATScJCCCGEmUgSFkIIIcxEknABzZw5k4oVK2JtbU1gYCB79uwxd0hF0pYtWwgJCcHLywuVSsWqVavMHVKRFB4eTqNGjXBwcKBMmTKEhoYSGxtr7rCKnFmzZlG3bl0cHR1xdHQkKCiItWvXmjusIu/zzz9HpVIxfPhwc4dS5EycOBGVSmX0ql69+lM7viThAliyZAkjRoxgwoQJHDhwAH9/f4KDg7l06ZK5Qyty0tPT8ff3Z+bMmeYOpUjbvHkzgwYNYteuXURGRpKdnc2LL75Ienq6uUMrUsqXL8/nn3/O/v372bdvH61ataJTp04cPXrU3KEVWXv37uWHH36gbt265g6lyKpVqxaJiYmG17Zt257ewRVhssaNGyuDBg0yfNZqtYqXl5cSHh5uxqiKPkBZuXKlucN4Jly6dEkBlM2bN5s7lCLPxcVFmTdvnrnDKJJSU1MVPz8/JTIyUmnRooUybNgwc4dU5EyYMEHx9/c32/HlTNhEWVlZ7N+/nzZt2hjK1Go1bdq0YefOnWaMTBQnN2/eBMDV1dXMkRRdWq2WxYsXk56eTlBQkLnDKZIGDRpEx44djX5fidxOnjyJl5cXlSpVokePHsTHxz+1Y5e4VZQe15UrV9BqtXh4eBiVe3h4cOLECTNFJYoTnU7H8OHDef7556ldu7a5wylyjhw5QlBQEBkZGdjb27Ny5Upq1qxp7rCKnMWLF3PgwAH27t1r7lCKtMDAQBYsWEC1atVITExk0qRJNGvWjJiYmKey4IUkYSGKmEGDBhETE/N070s9Q6pVq0Z0dDQ3b95k+fLlhIWFsXnzZknE9zh//jzDhg0jMjISa2trc4dTpLVv397wvm7dugQGBuLj48PSpUvp27fvEz++JGETubm5YWFhYViX+I7k5GTKli1rpqhEcTF48GD++usvtmzZQvny5c0dTpGk0WioUqUKAA0aNGDv3r18++23/PDDD2aOrOjYv38/ly5don79+oYyrVbLli1bmDFjBpmZmVhYWJgxwqLL2dmZqlWrcurUqadyPLknbCKNRkODBg2IiooylOl0OqKiouS+lCgwRVEYPHgwK1euZMOGDfj6+po7pGeGTqcjMzPT3GEUKa1bt+bIkSNER0cbXg0bNqRHjx5ER0dLAn6ItLQ0Tp8+jaen51M5npwJF8CIESMICwujYcOGNG7cmGnTppGenk7v3r3NHVqRk5aWZvQXZVxcHNHR0bi6ulKhQgUzRla0DBo0iIULF/Lnn3/i4OBAUlISAE5OTtjY2Jg5uqJjzJgxtG/fngoVKpCamsrChQvZtGkT69evN3doRYqDg0Ou8QR2dnaULl1axhncZ+TIkYSEhODj48PFixeZMGECFhYWdO/e/akcX5JwAXTr1o3Lly8zfvx4kpKSqFevHuvWrcs1WEvAvn37eOGFFwyfR4wYAUBYWBgLFiwwU1RFz6xZswBo2bKlUfn8+fPp1avX0w+oiLp06RI9e/YkMTERJycn6taty/r162nbtq25QxPPqAsXLtC9e3euXr2Ku7s7TZs2ZdeuXbi7uz+V48sqSkIIIYSZyD1hIYQQwkwkCQshhBBmIklYCCGEMBNJwkIIIYSZSBIWQgghzESSsBBCCGEmkoSFEEIIM5EkLIQQQpiJJGEhRKFRqVSsWrXK3GEI8cyQJCxEMdGrVy9UKlWuV7t27cwdmhDiAWTuaCGKkXbt2jF//nyjMisrKzNFI4R4FDkTFqIYsbKyomzZskYvFxcXQH+peNasWbRv3x4bGxsqVarE8uXLjfY/cuQIrVq1wsbGhtKlS9O/f3/S0tKM6vz000/UqlULKysrPD09GTx4sNH2K1eu0LlzZ2xtbfHz82P16tWGbdevX6dHjx64u7tjY2ODn59frj8ahChJJAkLUYKMGzeOLl26cOjQIXr06MHrr7/O8ePHAUhPTyc4OBgXFxf27t3LsmXL+Oeff4yS7KxZsxg0aBD9+/fnyJEjrF69mipVqhgdY9KkSXTt2pXDhw/ToUMHevTowbVr1wzHP3bsGGvXruX48ePMmjULNze3p9cBQhQ1ihCiWAgLC1MsLCwUOzs7o9enn36qKIqiAMqAAQOM9gkMDFQGDhyoKIqizJkzR3FxcVHS0tIM2yMiIhS1Wq0kJSUpiqIoXl5eykcfffTAGABl7Nixhs9paWkKoKxdu1ZRFEUJCQlRevfuXThfWIhiQO4JC1GMvPDCC4a1ie9wdXU1vA8KCjLaFhQURHR0NADHjx/H398fOzs7w/bnn38enU5HbGwsKpWKixcv0rp164fGULduXcN7Ozs7HB0duXTpEgADBw6kS5cuHDhwgBdffJHQ0FCaNGlSoO8qRHEgSViIYsTOzi7X5eHCYmNjk696pUqVMvqsUqnQ6XQAtG/fnnPnzrFmzRoiIyNp3bo1gwYN4quvvir0eIV4Fsg9YSFKkF27duX6XKNGDQBq1KjBoUOHSE9PN2zfvn07arWaatWq4eDgQMWKFYmKinqsGNzd3QkLC+O3335j2rRpzJkz57HaE+JZJmfCQhQjmZmZJCUlGZVZWloaBj8tW7aMhg0b0rRpU37//Xf27NnDjz/+CECPHj2YMGECYWFhTJw4kcuXLzNkyBDeeustPDw8AJg4cSIDBgygTJkytG/fntTUVLZv386QIUPyFd/48eNp0KABtWrVIjMzk7/++svwR4AQJZEkYSGKkXXr1uHp6WlUVq1aNU6cOAHoRy4vXryYd955B09PTxYtWkTNmjUBsLW1Zf369QwbNoxGjRpha2tLly5dmDp1qqGtsLAwMjIy+Oabbxg5ciRubm68+uqr+Y5Po9EwZswYzp49i42NDc2aNWPx4sWF8M2FeDapFEVRzB2EEOLJU6lUrFy5ktDQUHOHIoT4j9wTFkIIIcxEkrAQQghhJnJPWIgSQu48CVH0yJmwEEIIYSaShIUQQggzkSQshBBCmIkkYSGEEMJMJAkLIYQQZiJJWAghhDATScJCCCGEmUgSFkIIIcxEkrAQQghhJv8PprMItsj3ae0AAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 500x300 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "epochs_tensor = torch.linspace(0, num_epochs, len(train_accs))\n",
    "examples_seen_tensor = torch.linspace(0, examples_seen, len(train_accs))\n",
    "\n",
    "plot_values(epochs_tensor, examples_seen_tensor, train_accs, val_accs, label=\"accuracy\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "880a3791",
   "metadata": {},
   "source": [
    "- 根据上方的准确率图表可见，模型在第4和第5轮训练后达到了较高的训练及验证准确率；\n",
    "- 但需注意，我们在训练函数中设置了eval_iter=5，这意味着当前展示的性能指标是基于采样估算的结果；\n",
    "- 我们可以按以下方式在整个数据集上计算训练集、验证集和测试集的完整性能表现：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "id": "8e27c344",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training accuracy: 97.21%\n",
      "Validation accuracy: 97.32%\n",
      "Test accuracy: 95.67%\n"
     ]
    }
   ],
   "source": [
    "train_accuracy = calc_accuracy_loader(train_loader, model, device)\n",
    "val_accuracy = calc_accuracy_loader(val_loader, model, device)\n",
    "test_accuracy = calc_accuracy_loader(test_loader, model, device)\n",
    "\n",
    "print(f\"Training accuracy: {train_accuracy*100:.2f}%\")\n",
    "print(f\"Validation accuracy: {val_accuracy*100:.2f}%\")\n",
    "print(f\"Test accuracy: {test_accuracy*100:.2f}%\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "861497a6",
   "metadata": {},
   "source": [
    "- 可见训练集与验证集的性能表现几乎一致；\n",
    "- 但根据略低的测试集性能可知，模型对训练数据存在极轻微过拟合，同时对用于调整超参数（如学习率）的验证数据也存在轻微过拟合；\n",
    "- 这种现象属于正常情况，通过增加模型丢弃率（drop_rate）或优化器中的权重衰减（weight_decay）参数可能进一步缩小该差距；\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "590fc027",
   "metadata": {},
   "source": [
    "## 使用LLM作为垃圾邮件分类器"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "353117b4",
   "metadata": {},
   "source": [
    "- 最后，让我们实际部署微调后的GPT模型；\n",
    "- 下方的classify_review函数实现了与先前SpamDataset类似的数据预处理流程；\n",
    "- 该函数首先返回模型预测的整数类别标签，随后将其映射为对应的类别名称；\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "id": "84b17fd1",
   "metadata": {},
   "outputs": [],
   "source": [
    "def classify_review(text, model, tokenizer, device, max_length=None, pad_token_id=50256):\n",
    "    model.eval()\n",
    "\n",
    "    input_ids = tokenizer.encode(text)\n",
    "    supported_context_length = model.pos_emb.weight.shape[0]\n",
    "\n",
    "    input_ids = input_ids[:min(max_length, supported_context_length)]\n",
    "\n",
    "    input_ids += [pad_token_id] * (max_length - len(input_ids))\n",
    "    input_tensor = torch.tensor(input_ids, device=device).unsqueeze(0)\n",
    "\n",
    "    with torch.no_grad():\n",
    "        logits = model(input_tensor)[:, -1, :]\n",
    "    predicted_label = torch.argmax(logits, dim=-1).item()\n",
    "\n",
    "    return \"spam\" if predicted_label == 1 else \"not spam\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "id": "afad33a6",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "spam\n"
     ]
    }
   ],
   "source": [
    "text_1 = (\n",
    "    \"You are a winner you have been specially\"\n",
    "    \" selected to receive $1000 cash or a $2000 award.\"\n",
    ")\n",
    "\n",
    "print(classify_review(\n",
    "    text_1, model, tokenizer, device, max_length=train_dataset.max_length\n",
    "))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "id": "8444a91a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "not spam\n"
     ]
    }
   ],
   "source": [
    "text_2 = (\n",
    "    \"Hey, just wanted to check if we're still on\"\n",
    "    \" for dinner tonight? Let me know!\"\n",
    ")\n",
    "\n",
    "print(classify_review(\n",
    "    text_2, model, tokenizer, device, max_length=train_dataset.max_length\n",
    "))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "id": "86e76ab7",
   "metadata": {},
   "outputs": [],
   "source": [
    "torch.save(model.state_dict(), \"review_classifier.pth\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "id": "7f9f49ea",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<All keys matched successfully>"
      ]
     },
     "execution_count": 93,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_state_dict = torch.load(\"review_classifier.pth\", map_location=device, weights_only=True)\n",
    "model.load_state_dict(model_state_dict)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "build-LLM-from-scratch",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
